A proxy server enables end users on a Local Area Network (LAN) to access Web servers through a common gateway. In addition, the proxy server maintains a cache of the most recent requests from clients. If a server receives a request for information that is already stored in the cache, the server can quickly serve the request without incurring the overhead to search the Web.
Squid is an open-source Web proxy server, which can be configured on UNIX and Linux systems. Squid supports various protocols, such as Hypertext Transfer Protocol (HTTP), File Transfer Protocol (FTP), Independent Content Provider (ICP), Cache Array Routing Protocol (CARP), and Secure Sockets Layer (SSL). Squid has extensive access control mechanisms, which are maintained using Access Control Lists (ACLs). The Squid proxy server provides various logs for analyzing the access statistics.
This ReferencePoint describes the features, installation process, and configuration of the Squid proxy server on Linux. In addition, it explains the concept of cache management and describes some tips to troubleshoot a Squid proxy server.
Overview of Squid
The primary limitation with proxy servers, such as Microsoft Proxy Server and Apache, is that they do not support ICP, which is the fastest communication protocol currently available. The Squid proxy server for Linux uses ICP, which enhances its performance as compared to other proxy servers.
Introducing Web Proxy Servers
Web servers provide content from the Internet to numerous end users. As a result, these servers experience bandwidth and latency problems. Bandwidth is the data transfer rate, which determines the speed of an Internet connection. Latency is the time required to retrieve and send the requested Internet object, such as file, document, or response to queries, from the Web server to the client. Quite often, requests are for similar objects and repeated execution of similar requests results in repeated downloads of similar objects.
To overcome the problem of downloading similar Internet objects repeatedly, most browsers implement caching. Caching refers to storing the information of recently visited Internet objects in either the memory or the hard disk of a computer. A computer that stores and shares recently requested documents with clients is called a Web proxy server. A limited number of clients can access the cache maintained by a browser, but the cache in a proxy server is available to all the computers in a local network.
Proxy servers use the parent-child relationship to implement caching. The child can request any Internet object present in the parent proxy server. Different Web proxy servers can communicate using ICP, which is used to search requested documents in the neighboring Web caches within the hierarchy.
| Note | ICP should be used only when sibling and parent proxy caches exist. Otherwise, the process of using ICP increases the latency. |
System Requirements to Install Squid
Squid caches FTP and HTTP Uniform Resource Locators (URLs), documents, and Domain Name Server (DNS) lookups. It handles all the requests from clients in a single input/output (I/O)-driven process and supports encrypted transmission using various encrypted protocols, such as SSL, Secure Hypertext Transfer Protocol (HTTPS), and Transport Layer Security (TLS). Squid also provides the negative caching feature, which stores all Not Found and Connection Refused error messages for a period of five minutes.
The Squid proxy server consists of DNSSERVER, which is a DNS lookup program and UNLINKD, which is a program that deletes files from the cache directory. Squid can be installed on several operating systems including UNIX, Linux, FreeBSD, NetBSD, BSDI, and SunOS or Solaris. Before selecting the hardware to install Squid, you must take four key factors into account: disk random seek time, available system memory, sustained disk throughput, and processing power.
| Note | Seek time refers to the time taken to move the head of a hard disk from one random track to another. While installing Squid, a hard disk with a faster seek time must be used. |
The minimum requirements to install Squid are:
-
300 Megahertz (MHz) Pentium II CPU
-
512 Megabyte (MB) RAM
-
Four 9 Gigabyte (GB) Ultra Wide Small Computer System Interface (UW-SCSI) hard disks
-
A fully functional Transmission Control/Internet Protocol (TCP/IP) network with DNS configured
Installing Squid on Linux
Linux 7.1 and later versions have built-in support for Squid. To check if the Squid Red Hat Package Manager (rpm) package file is installed on the system, use the following command:
rpm –q squid
If the Squid rpm is already installed on a system, the following message is displayed:
Squid-2.4.STABLE2-2
If the Squid rpm is not installed on a system, the following message is displayed:
Package squid is not installed
To install Squid, insert the Squid installation CD and execute the following commands:
mount /mnt/cdrom
rpm –ivh
/mnt/cdrom/Redhat/RPMS/squid-2.3.STABLE2-2.t386.rpm
The installation procedure creates the Squid directory tree, as shown in Figure 4-4-1:
In Figure 4-4-1, the bin directory stores binary files and associated Squid tools. The cache directory stores cached objects. The etc directory stores Squid configuration files, and the src directory stores the source files for Squid programs.
Configuring Squid
To configure Squid, only a few parameters need to be modified in the squid.conf file located in the /etc/squid/ directory. All other parameters accept default values. To configure Squid, you need to configure the following parameters:
-
http_port: Specifies the HTTP port on which Squid listens to client requests. The default port number is 3128. Any number above 1024 can be specified as the HTTP port.
Note Port numbers below 1024 are reserved ports, also called known ports. These ports are meant for services such as HTTP, Simple Mail Transfer Protocol (SMTP), Network News Transfer Protocol (NNTP), and Post Office Protocol (POP).
A Squid proxy server can contain multiple ports to address client requests. To assign multiple ports, append the new port number to the existing port number using the following command:
http_port 3129 4000
Where, 3129 is the existing port number and 4000 is the new port number.
-
cache_dir: Specifies the path for the cache directory in the squid.conf file. To store cached documents in a directory in Squid, specify the size of the main directory in MB and the number of subdirectories in it using the following command:
cache_dir /usr/local/squid/cache 200 10 259
Where, 200 specifies the size of the cache directory in MB and the values 10 and 259 specify the number of first and second level subdirectories under the main cache directory.
-
cache_mgr: Specifies the e-mail address of the administrator to which a mail should be sent if the Squid daemon process dies and the proxy server stops responding. This e-mail address is included in the error messages that are displayed to end users.
Configuring FTP and HTTP Settings
Squid supports various protocols for Internet access, including FTP, which helps transfer files over the Internet for authenticated clients. FTP also supports anonymous end user logins, which do not require the end user to supply a password to access files.
Squid enables the administrator to specify an e-mail address as the value for the ftp_user parameter. This e-mail address is used as the password by the end user who logs on to the FTP server.
The default configuration of Squid can also be modified using a graphical user interface (GUI). To configure the default options of Squid:
-
Open the console window of Squid and type netconf on the command line. The Network configurator window appears.
-
Click the Server tasks tab, as shown in Figure 4-4-2:
-
Click Squid (http proxy + cache). The server displays the Squid administration dialog box, as shown in Figure 4-4-3:
 "Click To expand")
-
Click Defaults. The Squid defaults window appears.
-
Enter information about the HTTP port on which Squid receives requests from the clients and the size of the disk and RAM cache in MB. Figure 4-4-4 shows the Squid defaults window with the values specified:
 "Click To expand")
-
Click Accept.
Access Control Lists and Access Control Parameters
Squid provides a basic level of access control by default. For multi-user environments, such as an Internet Service Provider (ISP), where each end user has limited access, configure Squid to filter protocols, commands, domains, and routers.
Configuring access controls in a Squid server involves configuring ACL elements and Access Lists (ALs), which are available in the squid.conf configuration file. An AL contains the allow and deny keywords and ACL elements. The allow keyword instructs the Squid server to allow access to the URLs specified in the ACL whereas the deny keyword specifies that the URLs in the ACL should be denied access.
Table 4-4-1 lists the ACL elements present in the squid.conf file:
| ACL Element | Description |
|---|---|
| src | Defines the IP addresses of the clients. |
| dst | Defines the IP addresses of the destination servers. |
| myip | Defines the IP address of the local client. |
| time | Defines the time of the day and the day of the week. |
| url_regex | Defines the URL regular expression for pattern matching. All the URLs that match the regular expression will be allowed access. |
| urlpath_regex | Defines the path of the URL regular expression for pattern matching. This option excludes the protocol and host name. The URL path that matches the regular expression will be provided access. |
| port | Defines the port number of the server. |
| myport | Defines the local port number of the client. |
| srcdomain | Defines the domain name of the client. |
| dstdomain | Defines the domain name of the server. |
| srcdom_regex | Defines the expression that matches the name of the source computer or the domain name of the server. |
| dstdom_regex | Defines expression that matches the domain name of the destination computer or client. |
| proto | Defines the protocol, either HTTP or FTP, to be used for transferring data. |
| method | Defines the methods, such as Post and Get, to be used for HTTP requests. |
| browser | Defines the expression that matches the header of the browser. |
| ident | Matches the string of the end user name. |
| ident_regex | Used to define the regular expression to match the end user’s name. |
| src_as | Defines the Autonomous System Number (ASN) of the client. |
| dst_as | Defines the ASN of the destination server. |
| proxy_auth | Describes end user authentication through external programs, such as an application requiring access to the Proxy. |
| proxy_auth_regex | Describes end user authentication through external processes. This option is the same as proxy_auth except that the regular expression is used to obtain the names of the end users to be authenticated. |
| snmp_community | Describes string matching for Simple Network Management Protocol (SNMP) community. |
| maxconn | Defines the maximum number of connections allowed from a single client IP address. |
| req_mime_type | Defines the regular expression pattern matching on the mime type of the request. For example, acl aclname req_mime_type jpeg will search for jpeg in the mime header of the request. |
| arp | Matches Ethernet (MAC) addresses using Address Resolution Protocol (ARP) specified by this option. |
Open table as spreadsheetAll ALs cannot contain all ACL elements. Table 4-4-2 lists the ALs available in the squid.conf file:
| AL | Description |
|---|---|
| http_access | Allows access to or restricts HTTP clients from accessing the HTTP port of the Squid Server. |
| icp_access | Allows or denies queries to be passed to the Squid server from clients using ICP. |
| miss_access | Forces the neighboring proxy servers to use the current server as a sibling instead of a parent. This means that only the local clients are allowed to fetch misses and all other clients can only fetch hits. |
| no_cache | Defines responses that are not cached. |
| redirector_access | Controls access to the redirector pool, which is a pool of URLs to which queries are directed, and decides which requests will pass through it. |
| ident_lookup_access | Controls which requests need an Ident lookup. |
| always_direct | Used to specify the requests that are forwarded directly to servers from which the requests originated. |
| never_direct | Controls the requests that are not forwarded directly to servers from which the request originated. |
| snmp_access | Controls the access of SNMP clients to the cache. |
| broken_posts | Defines requests for which Squid appends an extra Carriage Return followed by Line Feed (CRLF) character after POST message bodies. This CRLF is required by some origin servers, which have broken implementation of PUT/POST requests. |
| cache_peer_access | Determines which requests are forwarded to a specific neighboring proxy. |
To enable end users to use the cache maintained in a Squid server, use the following command:
acl myclients src 192.168.5.0
http_access allow myclients
To restrict the requests from a specific server to be cached, use the following command:
Acl myServer mydomain .newServer.com
No_cache deny abcServer
To put a ban on the terms, such as games and movies, in an ACL list of URLs, use the following code statements:
Acl banword1 url_regex games
Acl banword2 url_regex movies
http_access deny banword1
http_access deny banword2
http_access allow all
To block specific end users and user groups from accessing the server cache, use the following code statements:
Ident_lookup on
Acl myGroup user Jacque Zoe Marylou Richard Vera
http_access allow myGroup
http_access deny all
Communicating with Other Proxy Servers
Squid supports a hierarchy of proxy servers, which means that Squid searches these proxy servers in a hierarchical order. When requesting content from proxy servers, Squid does not send a request to the proxy servers in the order they appear in the hierarchy. It sends the request to all the servers together. After receiving the requested object from a proxy server, the Squid server sends it to the client browser.
The following code configures the cache_peer parameter in the squid.conf file to specify the proxy servers with which the Squid server communicates to retrieve objects from the Internet:
cache_peer parent.ald.net parent 3128 3130
cache_peer sib1.ald.net sibling 8080 3130
cache_peer sib2.ald.net sibling 8080 3130
The first line in this code specifies that the Squid server communicates with the parent server on port 3128 and uses ICP to query the server on port 3130. The second and third lines configure Squid to handle HTTP requests on port 8080 and ICP requests on port 3130.
To configure Squid to bypass other servers in the hierarchy and communicate directly with the parent server, insert the keyword parent with the cache_peer parameter in each line of the code.
Running Squid
Before running the Squid server, you need to create the swap directories that store the cache swap files. Use the following command to create the swap directories:
% /usr/local/squid/bin/squid –z
After creating the swap directories, start the Squid server process from the command prompt in the debugging mode using the following command:
% /usr/local/squid/bin/squid –NCd1
If the process starts successfully, the following message is displayed:
Ready to serve requests
To run Squid as a background daemon process, type the following command:
% /usr/local/squid/bin/squid
To check if the Squid server is running, type the following command:
% squid –k check
Managing Cache in Squid
Squid uses several protocols, such as ICP, Cache Digests, Hypertext Caching Protocol (HTCP), and CARP to communicate with other caches and proxy servers.
ICP is used to communicate between Squid caches. It is used to search for duplicate Web objects in caches. If the search locates a similar object, a hit message is returned. Otherwise, the search returns a miss message. ICP is currently implemented over Universal Datagram Protocol (UDP).
HTCP uses request and response headers for cache management. A request header contains basic information about a request, such as the name of the Web object, its address, and the name of the client agent. A response header contains basic information about the response, such as the location of the object. HTCP also enables you to monitor additions and deletions from a remote cache.
CARP is a peering protocol provided by Microsoft. It is used to build redundant arrays of small proxies as a single logical cache. Hash-based routing determines the best path to retrieve a specific Web object from an array. CARP provides information about an object, such as its location and whether or not a request for an object is the first request. CARP also provides a deterministic path or the optimum path to reach a specified URL. CARP avoids query messaging between proxy caches. It eliminates duplication of Web objects that would otherwise occur in an array of caches.
Configuring and Selecting Peer Servers
If in a large network were to use only one cache server, it would get overloaded and slow down access. In Squid, proxy cache servers communicate with other cache servers, called peers, in a hierarchy. The result of this hierarchy is improved latency.
To configure a peer server in Squid, modify the cache_peer, cache_peer_domain, neighbor_type_domain, icp_query_timeout, maximum_icp_query_timeout, dead_peer_timeout, mcast_icp_query, hierarchy_stoplist, and no_cache parameters in the squid.conf file.
The cache_peer Parameter
Configuring the cache_peer parameter in the squid.conf file enables the Squid server to communicate with its peers. This parameter specifies basic information, such as the host name and port number to which queries must be sent. The syntax to configure the cache_peer parameter is:
cache_peer
Where, host-name specifies the host name of the cache server that is queried, type describes the relationship between the server and its peers, http-port indicates the port number of the destination server, icp-port specifies the port number for ICP, and options specifies various peer options available.
Table 4-4-3 lists the various peer options available for the cache_peer parameter:
| Peer Option | Description |
|---|---|
| Proxy-only | Specifies that the data retrieved from the remote cache must be stored locally. By default, Squid stores data locally to improve latency, but this process results in wasted disk space. |
| Weight=n | Specifies a weight for retrieving a Web object from a cache server when the object is present in multiple servers. Higher values are prioritized. The default value is 1. |
| Ttl=n | Specifies an IP multicast Time To Live (TTL) value, which is used while sending ICP queries to multicast groups. |
| No-query | |
| Default | Used when a parent cache is specified as the last resort and ICP is disabled. |
| Login=user:password | Specifies the user name and password when the parent server requires proxy authentication. |
The cache_peer_domain Parameter
The cache_peer_domain parameter specifies the list of domains for which neighboring caches are queried. An exclamation mark before the domain name specifies that the cache is queried for objects that do not exist in that domain. The domain names may be on the same line or different lines.
When ACL contains multiple domains for a particular cache host, the first matching domain is selected. If no domains are specified for a particular host, the host is queried for all requests. The syntax to use the cache_peer_domain parameter is:
cache_peer_domain
UDP query packets are sent to the parent server only when the requested object exists on a server in the org domain.
The neighbor_type_domain Parameter
The neighbor_type_domain parameter is used to modify the neighbor type for specific domains. The syntax to use this parameter is:
neighbor_type_domain parent | sibling domain [domain...]
The icp_query_timeout Parameter
The ICP query timeout value specifies the time for which a Squid server waits for the response to a specific request. Squid, based on the round-trip-time of recent ICP queries, automatically determines this value. Round-trip-time refers to the time interval between the first byte of an HTTP request that is sent and the last byte of the server response that is received. The default value is zero, which can be overridden.
The syntax to use the icp_query_timeout parameter is:
icp_query_timeout
The maximum_icp_query_timeout Parameter
Determining the ICP query timeout value dynamically may produce large values, , such as five seconds, as a result. The maximum_icp_query_timeout parameter sets an upper limit for the query timeout. The syntax to use this parameter is:
maximum_icp_query_timeout
The dead_peer_timeout Parameter
This parameter specifies the time period for which Squid waits before declaring a peer as dead. The default value is 10 seconds. The syntax to use this parameter is:
dead_peer_timeout
The mcast_icp_query_timeout Parameter
Squid sends out ICP probes at regular intervals for multicast peers. An ICP probe is an ICP request sent out to test the connection to a particular computer. These probes count the number of other peers listening on a particular multicast address. The value of the mcast_icp_query_timeout parameter specifies the number of milliseconds Squid should wait before counting all the replies. The syntax to use this parameter is:
Mcast_icp_query_timeout
The hierarchy_stoplist Parameter
The hierarchy_stoplist parameter provides a list of words to the server. When the requested URL contains any of these words, the cache handles the object directly. The following statement is an example:
Heirarchy_stoplist cgi_bin ?
The above code line indicates that all the URLs containing cgi-bin or ? will not go through cache peers and will be retrieved from the origin servers directly.
The syntax to use the hierarchy_stoplist parameter is:
hierarchy_stoplist
If an ASP or JSP is found in the requested URL, Squid queries the caches. Because these pages are dynamic, Squid accesses the object directly from the source.
Cache Digests and Cache Hierarchy Structures
Cache digests contain information in a compressed format about all the Internet objects in a cache server. The information is stored in a hash table to indicate if the requested URL is present in the cache. As a result, Squid can locate Internet objects without utilizing system resources or searching the disk.
Squid checks this list of Internet objects periodically and converts the information into a bitmap, which is then transferred across the network. If the bit in the map is on, it indicates that the Internet object is available in the cache. If it is off, it indicates that the object is not present. Other cache servers can also use this bitmap when they connect to the Squid server on the HTTP port.
| Note | Although the cache digest is generated automatically every ten minutes, it can be configured manually also. |
ICP packets contain only the URLs of requested Internet objects and not the header information. This lack of information may cause the Squid server to serve wrong pages. This problem is addressed using cache digests because they use both headers and URLs.
Cache digests are based on the boom-filters method, which represents a set of keys with lookup capabilities, which check if the key is in the filter. In this method, a one-dimensional array of m bits is allocated, with a value of zero assigned to each bit. A k number of independent hash functions are selected, which are denoted by h1, h2,...hk. A set of n keys are denoted by A = {a1, a2, a3... an}.
The capacity of the cache digest is determined using the following equation:
Number of bits in the array = capacity * bits_per_entry + 7
The size of the digest in bytes is specified using the following formulae:
int (number of bits in an array /8)
A change in the cache size is measured when a digest rebuild occurs. Squid determines the object to be added to the list by checking if the object is private, negatively cached, or not cacheable and the expiry time. If any of these checks returns a positive result, that object is not added to the list.
Squid Log Files
Squid log files contain information about the workload and other troubleshooting information, such as access information, errors, and memory and resource utilization. Squid uses multiple log files to store several types of logs.
The various log files stored in the Squid server are:
-
Squid.out: Maintains a log of the startup times. It also maintains the list of fatal errors.
-
Cachelog: Contains the error messages generated by Squid. This includes errors that occur while installing new features on a Squid server.
-
Useragent.log: Contains information about the distribution of client browsers.
-
Store.log: Contains information about objects currently or previously stored on the hard disk. This information is used while debugging.
-
Access.log: Contains information about access to the proxy server.
| Note | Do not delete the access.log, store.log, and cache.log files while Squid is running because they cannot be recovered later. |
You can use the rotate feature provided by Squid to maintain a log file. When this feature is turned on, the current log files are first closed and then reopened with a numeric extension such as .0 or .1, which separates the old log files from the new ones. Setting the logfile_rotate parameter to 0 closes the log files and reopens them.
Troubleshooting Tips
If you see the message Proxy Access Denied when trying to access the proxy cache server, check the http_access option in the squid.conf configuration file. For any errors in configuration information, check the access.log and squid.conf files.
To prevent local objects from being cached, use the cache_stoplist or http_stop configuration options in the squid.conf file. When a sibling cache is serving an object, the requesting client may not receive the response because objects move between the siblings and do not reach the client. This happens when the ICP port number is correct but the HTTP port number is wrong. Change the HTTP port number to resolve this problem.
The error message Too many open files is displayed because of a shortage of file descriptors, which specify the number of files that can be opened by Linux. This happens when the operating system has a low filedescriptors limit. This limit can be configured in the operating system kernel. For Linux kernel version 2.2.x, the maximum number of file handles is read and written by simply accessing file-max and inode-max files, as shown in the following code:
/proc /sys/fs/file-max
/proc/sys/fs/inode-max
To increase the filedescriptor limit, execute the following command:
Echo 3072 > /proc/sys/fs/file-max
If you see the error message Illegal characters in the hostname; underscores are not allowed, it is because Squid does not allow underscores in hostnames. To override the default settings, rerun the configuration file using the following code:
./configure –enable-underscores ...
and then recompile:
make clean
make
If you see the error message, The default parent option is not working, it indicates that all connections are not being forwarded to the parent. This default option is used only as a last resort. To override the default and force all requests to the parent caches, use the following command:
acl all src 0.0.0.0/0.0.0.0
never_direct allow all
Configuring Browsers
After installing the Squid proxy cache server, a client browser can be instructed to fetch objects from the proxy instead of from the Web. Most browsers today support proxy servers and can be easily configured for the Squid proxy server. For example, Netscape Navigator and Microsoft Internet Explorer (IE) support features, such as list domains and URL patterns, which are not fetched through the proxy.
To access the Internet using the Squid proxy server, specify the following in the client Web browser:
-
The host name of the proxy server to which the client connects.
-
The port on which the server listens client requests.
The host name is the name that identifies the proxy server. Use a proxy-specific host name that will be useful when the cache needs to be migrated to another computer. It is easier to change the DNS name rather than the configuration of every browser on the network.
Configuring Netscape Communicator
Netscape Communicator is available with Linux by default. To configure this browser and access the Web using Squid:
-
Open Netscape Communicator and select the Edit menu option.
-
Select Preferences to open the Preferences window.
-
Click the Advanced option to expand it.
-
Select the Proxies option. A new window appears in the adjacent panel, as shown in the Figure 4-4-5:
 "Click To expand")
-
Select the Manual proxy configuration radio button and click View. A new window appears.
-
Specify the proxy server information, as shown in Figure 4-4-6:
Figure 4-4-6: The Netscape View Manual Proxy Configuration Window
 "Click To expand")
-
Click OK to save the settings.
Specify the host name or IP addresses for FTP, Gopher, HTTP, Security, and WAIS in the appropriate text boxes. Specify a common port number, 3128, for all protocols. If the Squid server does not support a protocol, leave that field blank.
Configuring Internet Explorer
In addition to Netscape Communicator, you can install and configure Internet Explorer to access the Web using Squid.
To configure Internet Explorer for the Squid proxy server:
-
Open Internet Explorer. From the Tools menu select Internet Options. The Internet Options dialog box appears, as shown in Figure 4-4-7:
 "Click To expand")
-
In the Connections tab, click the LAN Settings button. The Local Area Network (LAN) Settings dialog box appears.
-
Click to select the Use a proxy server option.
-
Enter the host name of the Squid server in the Address field and the port number 3128 in the Port field, as shown in Figure 4-4-8:
Figure 4-4-8: Specifying the Address of the Squid Proxy Server
 "Click To expand")
-
Click the Advanced button to open the Proxy Settings dialog box.
-
Specify the appropriate host name or the IP address of the Squid server in the HTTP, FTP, and Gopher fields.
-
Specify the port number 3128 for HTTP, FTP, and Gopher protocols.
-
Click OK to save the settings.
