Uncategorized

Docker Nginx and Sentiment Engine on Steroids

Recipe for 74 Million request per day

In the blog post I will explain a battle tested setup which could let you scale http requests upto 860 req/s or a cummulative of 74Million requests per day.

Lets start with our requirements. We needed a low latency sentiment classification engine for serving literally millions of Social Mentions per day. Of late load against sentiment engine cluster has been increasing considerably after Viralheat’s pivot to serve Enterprise customers. The existing infrastructure was not able to handle the new load forcing us to have friday night out to fix it.

Setup

  • ) Nginx running on bare metal
  • ) Sentiment Engine powered by Torando Server in Docker instances. ( Docker version 0.7.5)

In a perfect world the default kernel setting should have worked for any kind of workload but in reality it wont work. The defaults kernel setting are not suitable for high load and are mainly for general purpose networking. In order to serve heavy short lived connections we need to modify/tune certain OS setting along with the tcp settings.

First increase the open file limit

Modify /etc/security/limits.conf to have a high number for open file descriptors. Since every open files takes some OS resources make sure you have sufficient memory don’t blindly increase the open file limits.

*               soft     nofile          100000
*               hard     nofile          100000

Sysctl Changes

Modify /etc/sysctl.conf to have these parameters.

fs.file-max = 100000
net.ipv4.ip_local_port_range = 2000 65000
net.ipv4.tcp_fin_timeout = 5
net.ipv4.tcp_keepalive_time = 1800
net.ipv4.tcp_window_scaling = 0
net.ipv4.tcp_sack = 0
net.ipv4.tcp_tw_recycle = 1
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_max_syn_backlog = 3240000
net.ipv4.tcp_max_tw_buckets = 1440000
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216
net.ipv4.tcp_congestion_control = cubic

net.core.rmem_default = 8388608
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
  • net.ipv4.ip_local_port_range Nginx need to create two connection for every request, one to the client and the other one to the upstream server. So increasing the port range will prevent for port exhaustion.
  • net.ipv4.tcp_fin_timeout The minimum number of seconds that must elapse before a connection in TIME_WAIT state can be recycled. Lowering this value will mean allocations will be recycled faster
  • net.ipv4.tcp_tw_recycle Enables fast recycling of TIME_WAIT sockets. Use with caution and ONLY in internal network where network connectivity speeds are “faster”.
  • net.ipv4.tcp_tw_reuse This allows reusing sockets in TIME_WAIT state for new connections when it is safe from protocol viewpoint. Default value is 0 (disabled). It is generally a safer alternative to tcp_tw_recycle. Note: The tcp_tw_reuse setting is particularly useful in environments where numerous short connections are open and left in TIME_WAIT state, such as web servers. Reusing the sockets can be very effective in reducing server load.

Make sure you run sudo sysctl -p after making modifications to the sysctl.conf.

NGINX Configurations

worker_processes  auto;
worker_rlimit_nofile 96000;
events {
  use epoll;
  worker_connections  10000;
  multi_accept on;
}

http {
        sendfile    on;
        tcp_nopush  on;
        tcp_nodelay on;

        reset_timedout_connection on;
}

upstream sentiment_server{
        server server0:9000;
        server server1:9001;
        server server2:9002;
        server server3:9003;
        server server4:9004;
        server server5:9005;
        server server6:9006;
        server server7:9007;
        server server8:9008;
        server server9:9009;
        server server10:9010;
        server server11:9011;
      keepalive 512;
}

server {
  server_name serverip;
  location / {

    proxy_pass http://senti_server;
    proxy_set_header   Connection "";
    proxy_http_version 1.1;
    break;
  }
}
  • worker_processes defines the number of worker processes that nginx should use when serving your website. The optimal value depends on many factors including (but not limited to) the number of CPU cores, the number of hard drives that store data, and load pattern. When in doubt, setting it to the number of available CPU cores would be a good start (the value “auto” will try to autodetect it).
  • worker_rlimit_nofile changes the limit on the maximum number of open files for worker processes. If this isn’t set, your OS will limit. Chances are your OS and nginx can handle more than ulimit -n will report, so we’ll set this high so nginx will never have an issue with “too many open files”.
  • worker_connections sets the maximum number of simultaneous connections that can be opened by a worker process. Since we bumped up worker_rlimit_nofile, we can safely set this pretty high.

References

Docker for sentiment engine.

Our sentiment engine runs inside a docker container which helps us in iterating and deploying new models fast. Our initial assumption was that running inside a docker would have performance overhead but it wasn’t. We tuned our container with similar configurations as the base machine. The sysctl.conf inside the container was almost similar to the host machine.

A good addition to the backend infrastructure would be some kind of a intelligent container which can look at the load and scale up or scale down the sentiment engine instances. This can be easily done as docker exposes a REST API to create and destroy the container on the fly. If you like interested with the work we do check our careers pageViralheat Careers

FYI Please do not copy paste these setting and assume it will work automatically. There are many variable like the server machine memory, cpu etc. This guide should be used to help you with tuning.

About the author

Shrikar

Backend/Infrastructure Engineer by Day. iOS Developer for the rest of the time.

/* ]]> */