Find the answer to your Linux question:
Page 1 of 2 1 2 LastLast
Results 1 to 10 of 11
Hello to all friends. How can I install cache server for caching downloaded files ? I try use Squid for it. but it's not working properly. Squid can caching downloaded ...
Enjoy an ad free experience by logging in. Not a member yet? Register.
  1. #1
    Just Joined!
    Join Date
    Mar 2013
    Posts
    5

    Question How to create a cache server to caching downloaded files (with IDM)


    Hello to all friends.
    How can I install cache server for caching downloaded files ?

    I try use Squid for it.
    but it's not working properly.
    Squid can caching downloaded files when download the file without Download Managers.(Like IDM)

    When I use a Download Manager , Squid can't cache the downloaded file. ( Max connections number in IDM = 16)

    Please help me , How can I cache downloaded file , when download file with IDM and multi connection mode (max connections number = 16 or 8 )

    Thanks.

  2. #2
    Just Joined!
    Join Date
    Mar 2013
    Posts
    5
    My download speed is 500 KB/s (When I'm downloading files without Internet Download Manager).
    If I use the IDM (Or any other Internet Download Manager) and set the connections number to 16 , increase my download speed to 800 KB/s.

    I want to downloading files with 800 KB/s speed.

    If I set the connections number to 1 , Squid or Polipo cache is working properly , but my download speed decrease to 500 KB/s.
    And if I set the connections number to 16 or any number that's greater than 1 , cache not working properly.

    My cache server only use with Internet Download Manager and it's not set in the web browsers.

  3. #3
    Just Joined!
    Join Date
    Mar 2013
    Posts
    5
    It cannot be done ?!!

  4. #4
    Linux Guru Lazydog's Avatar
    Join Date
    Jun 2004
    Location
    The Keystone State
    Posts
    2,677
    IDM works by breaking up the file into multiple parts. These parts are download simultaneously and after completing the download it rebuilds the complete file. Because of this, speculation on my part, squid see this as many different connections and no just one single connection thus the file doesn't get cached as you would expect.

    So the question is then what do you want? Speed or caching?

    I do not use a proxy so what I am saying could be wrong, but it sounds logical.

    Regards
    Robert

    Linux
    The adventure of a life time.

    Linux User #296285
    Get Counted

  5. #5
    Just Joined!
    Join Date
    Mar 2013
    Posts
    5
    Thank you very much.
    Ok,but there is no solution ?

    Is it possible that after completing the download , Squid rebuilds the complete file in proxy server ?!
    Is there any other solution other than Squid ?

    The employer has requested this project ! Not me ! and I'm forced to do it.

  6. #6
    Linux Guru Lazydog's Avatar
    Join Date
    Jun 2004
    Location
    The Keystone State
    Posts
    2,677
    I would be more interested in how IDM is bypassing the proxy. The proxy should be the one downloading the file not IDM. IDM should be receiving the file from the proxy.

    Regards
    Robert

    Linux
    The adventure of a life time.

    Linux User #296285
    Get Counted

  7. #7
    Just Joined!
    Join Date
    Mar 2013
    Posts
    5
    IDM's traffic is passing from proxy ( with multiple connections for one file ) , but Squid could not be combining the parts of one file and store it in cache.

    Can I rebuild the parts of downloaded file after completing download and save it in cache ?

  8. #8
    Linux Guru Lazydog's Avatar
    Join Date
    Jun 2004
    Location
    The Keystone State
    Posts
    2,677
    Only if you know the order of how the file is to be placed back together.

    Regards
    Robert

    Linux
    The adventure of a life time.

    Linux User #296285
    Get Counted

  9. #9
    Just Joined! kerim's Avatar
    Join Date
    May 2010
    Location
    Istanbul
    Posts
    15

    Post Advanced Cache Managment Squid Script Snipet

    I will sent a configuration of advanced caching mechanism based system.

    Add these lines to your squid.conf:

    Code:
    #
    # Add any of your own refresh_pattern entries above these.
    #
    refresh_pattern ^ftp:           1440    20%     10080
    refresh_pattern ^gopher:        1440    0%      1440
    refresh_pattern -i (/cgi-bin/|\?) 0     0%      0
    refresh_pattern .               0       20%     4320
    
    # Cache managment section
    
    # DENY CACHING FOR FILETYPES etc.
    acl NONECACHEDFILE urlpath_regex -i "/etc/squid/acl/cache/none-cached-file.acl"
    cache deny NONECACHEDFILE
    
    acl NONECACHEDMIME rep_mime_type -i "/etc/squid/acl/cache/none-cached-mime.acl"
    cache deny NONECACHEDMIME
    
    acl NONECACHEDSITE url_regex -i "/etc/squid/acl/cache/none-cached-site.acl"
    cache deny NONECACHEDSITE
    
    acl NONECACHEDMETHOD method "/etc/squid/acl/cache/none-cached-method.acl"
    cache deny NONECACHEDMETHOD
    
    acl NONECACHEDCLIENT src "/etc/squid/acl/cache/none-cached-client.acl"
    cache deny NONECACHEDCLIENT
    
    acl NONECACHEDSERVER dst "/etc/squid/acl/cache/none-cached-server.acl"
    cache deny NONECACHEDSERVER
    
    acl NONECACHEDDOMINCLIENT srcdom_regex -i  "/etc/squid/acl/cache/none-cached-domain-client.acl"
    cache deny NONECACHEDDOMINCLIENT
    
    acl NONECACHEDDOMAINSERVER dstdom_regex -i "/etc/squid/acl/cache/none-cached-domain-server.acl"
    cache deny NONECACHEDDOMAINSERVER
    
    acl NONECACHEDMAC arp "/etc/squid/acl/cache/none-cached-mac.acl"
    cache deny NONECACHEDMAC
    
    #acl NONECACHEDUSER ident "/etc/squid/acl/cache/none-cached-user.acl"
    #cache deny NONECACHEDUSER
    
    acl NONECACHEDBROWSER browser "/etc/squid/acl/cache/none-cached-browser.acl"
    cache deny NONECACHEDBROWSER
    
    acl NONECACHEDREFERER referer_regex -i "/etc/squid/acl/cache/none-cached-referer.acl"
    cache deny NONECACHEDREFERER
    
    hierarchy_stoplist cgi-bin ?
    acl QUERY urlpath_regex cgi-bin \?
    cache deny QUERY
    Example acl files:

    none-cached-domain-server.acl;
    Code:
    \.goodsync\.com
    \.google\.com
    \.gmail\.com
    \.yandex\.com
    \.bing\.com
    none-cached-mime.acl:
    Code:
    ^text/.*
    ^image/.*
    ^audio/.*
    ^message/.*
    ^video/.*
    application/x-shockwave-flash
    x-ms-asf
    mms
    ms-hdr
    x-fcs
    none-cached-referer.acl:
    Code:
    \.sf\.net
    \.sourceforge\.net
    Above, These are exception caching acl rules.

  10. #10
    Just Joined! kerim's Avatar
    Join Date
    May 2010
    Location
    Istanbul
    Posts
    15
    Fill other acl files with your decision.

    i will add another commands for missing acl files and directory

    Code:
    mkdir -p /etc/squid/acl/cache
    touch /etc/squid/acl/cache/none-cached-file.acl
    touch /etc/squid/acl/cache/none-cached-mime.acl
    touch /etc/squid/acl/cache/none-cached-site.acl
    touch /etc/squid/acl/cache/none-cached-method.acl
    touch /etc/squid/acl/cache/none-cached-client.acl
    touch /etc/squid/acl/cache/none-cached-server.acl
    touch /etc/squid/acl/cache/none-cached-domain-client.acl
    touch /etc/squid/acl/cache/none-cached-domain-server.acl
    touch /etc/squid/acl/cache/none-cached-mac.acl
    touch /etc/squid/acl/cache/none-cached-user.acl
    touch /etc/squid/acl/cache/none-cached-browser.acl
    touch /etc/squid/acl/cache/none-cached-referer.acl
    Fill your acls as your desire

    All of them for none cached contents

Page 1 of 2 1 2 LastLast

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •