Write a web server in Common Lisp part one
Not so long ago I started learning Common Lisp. As it may seem, learning a new programming language — it is very not easy, especially if it is quite unlike all those languages, which had to face earlier. So I decided to start with the book Land Of Lisp. The book is very good, with interesting pictures and is very well suited for beginners. In one of the chapters was how to create a web server in Common Lisp. I decided to develop this theme, and in the end I turned out not quite what was described in this Chapter, and a very interesting web server. Source code you can see here.
For his writing, we need Linux installed emacs, sbcl, slime and quicklisp. Describe how it is to install, configure and how to use it, I wouldn't — in the Internet there are many articles about it. Our entire web server will be in one package, called myweb. Create a folder with the given name, and in it create two folders log and web. The log folder will contain the log file of the web server. In the web folder will be the html pages and images that the web server will give to clients. All the web server consists of seven files.
Start with the file, objavljamo package and asd file describing the package.
Create the package file.lisp:
the
As you can see, our web server consists of three packages:
the
Function in-package is usually placed at the beginning of the file and specifies the name of the package in which we declare variables and functions. In this case, as we announce the packages, we need to declare them in the main package :cl-user.
Notice the Directive use and export in the advertisement packages. use allows us to use functions from other packages without specifying the package names in the beginning of the function name, thereby reducing the amount of typing you have. export specifies the names of the functions can be used outside the package. As you can see, we have in the package :myweb is the function :start-and http :stop-http. While in package cl-user, we will not be able to call them via myweb:start-http if not pre-declare them using the Directive :export.
The advertisement packages we have, it now remains to write the actual source code of these packages. Create a file web.lisp, util.lisp and handler.lisp and each of them add a call to in-package. For the web.lisp (in-package :myweb), for util.lisp (in-package :myweb.util), etc. We will also need to create the log file.c calling lisp (in-package :cl-log). This file is required for startup and configuration of the logging system cl-log.
Finishing touch create a file structure for the web server will generate file myweb.asd, which describes which files the system should load the asdf to have worked.
the
Key :serial t indicates that asdf has uploaded files in the same order in which they are listed.
Now we need to write the file load.the lisp that will load our package and start the swank server for slime.
the
To continue development, we need to start swank and load all required libraries using quicklisp. To do this, run sbcl, being in the myweb directory, and call the function (quicklisp:quickload "swank"). After installing swank launch a swank server, by calling (swank:create-server) from the command line sbcl.
Let's start with the web server. For him, we need sockets. To work with sockets, I decided to implement with greatly expanded performance class libraries usocket. Also, we need threads (threads), for which we will use bordeaux-threads. But first I would like to tell you about the model of processing http requests we are going to create. Each request is processed by a separate thread. We will have threads-worker s to be created depending on the number of requests. Among them we will have a separate idle-threads that after completion of the processing of the request will go to the state of condition-wait, waiting for new requests. It is thus possible to reduce the load from creating new worker threads. It is a kind of mechanism thread pool for processing http requests.
Let's start with the announcement of socket and mutex variables for s in web.lisp:
the
To accept and distribute requests for flows we will use a separate thread, a pointer to which is stored in the *listen-thread*. Let's start with the start method-http:
the
It's a simple function to start the thread-dispenser, which in turn will call the function http-acceptor. We also have two key workers-limit — maximum number of worker s, and idle-workers — number of idle worker-ov.
Write the actual function of the distribution of the queries:
the
The first thing we do is socket-listen on the specified address and port. Further in a cycle we do the socket-accept, resulting in socket on the connected client that we have to process to the worker. Plus, we next request request-id. At this stage we must decide what to do with the request and how to handle it. First, we check the number of idle threads. If we have all the worker's are busy, we add the request to the queue for processing. If we have a free idle worker, then we again add the request in a queue, but this time caused (condition-notify (caar *idle-workers*))). And in the third case, we simply create new worker and send him a request that will be processed in the function worker-thread. Everything is quite simple. Only need to write the function processing of the worker thread:
the
If we had a call with a request-id, we need to first handle the request. We simply call the auxiliary function http-worker and pass it the client socket. Next we check that there is still processing requests: just remove the first request from the queue and pass it to worker thread for processing, thereby the function of the worker-thread recursively. The question may arise "and whether there will be recursion limit from the fact that the stack overflows at some point, for example, for a large number of requests in the queue?" So after calling a worker-thread recursively we did in the function is not called, the recursion limit is not going to happen. Almost all contemporary Common Lisp implementation supports this optimization. Well, if the queue is empty, then we have to check the number of idle worker-ov. If we have everything in order, we simply complete the request and remove the worker from the list worker-ov. If not, then we make the condition-wait, and thus the worker becomes idle worker.
If you noticed, we also call the list-workers. This helper function, which simply clears the worksheet worker s from dead threads.
Left to write http-worker function:
the
Here we create a socket-stream, parsim request and pass it as myweb.handler:process-request (about these functions we will discuss in the second part). list-workers simply returns us the list worker-ov, pre-clearing it from dead threads. We call this function in worker-thread before condition-wait.
The last thing we need to do is to write a stop function-http, which will stop our web server:
the
As you can see, it's simple — we stop the flow dispenser, kill all worker-s and reset the list.
And so, all ready to handle our requests. We'll talk about that in second part.
Thank you for your attention!
PS Thanks ertaquo for help with spelling and layout-ohms
Article based on information from habrahabr.ru
For his writing, we need Linux installed emacs, sbcl, slime and quicklisp. Describe how it is to install, configure and how to use it, I wouldn't — in the Internet there are many articles about it. Our entire web server will be in one package, called myweb. Create a folder with the given name, and in it create two folders log and web. The log folder will contain the log file of the web server. In the web folder will be the html pages and images that the web server will give to clients. All the web server consists of seven files.
Start with the file, objavljamo package and asd file describing the package.
Create the package file.lisp:
the
(in-package :cl-user)
(defpackage :myweb
(:use :cl :usocket :bordeaux-threads)
(:export :start-http :stop-http :list-workers :list-requests))
(defpackage :myweb.util
(:use :cl :local-time)
(:export :parse-request :read-utf-8-string :response-write :get-param :get-header :http-response :file response :html-template :log, info :log-warning :log-error))
(defpackage :myweb.handler
(:use :cl)
(:export :process-request))
As you can see, our web server consists of three packages:
the
myweb — will contain functions to start and stop the web server
myweb.util — will contain functions that help to handle requests
myweb.handler — will contain the code of request processing
Function in-package is usually placed at the beginning of the file and specifies the name of the package in which we declare variables and functions. In this case, as we announce the packages, we need to declare them in the main package :cl-user.
Notice the Directive use and export in the advertisement packages. use allows us to use functions from other packages without specifying the package names in the beginning of the function name, thereby reducing the amount of typing you have. export specifies the names of the functions can be used outside the package. As you can see, we have in the package :myweb is the function :start-and http :stop-http. While in package cl-user, we will not be able to call them via myweb:start-http if not pre-declare them using the Directive :export.
The advertisement packages we have, it now remains to write the actual source code of these packages. Create a file web.lisp, util.lisp and handler.lisp and each of them add a call to in-package. For the web.lisp (in-package :myweb), for util.lisp (in-package :myweb.util), etc. We will also need to create the log file.c calling lisp (in-package :cl-log). This file is required for startup and configuration of the logging system cl-log.
Finishing touch create a file structure for the web server will generate file myweb.asd, which describes which files the system should load the asdf to have worked.
the
;; myweb.asd
(asdf:defsystem #:myweb
:serial t
:components ((:file "package")
(:file "log")
(:file "util")
(:file "web")
(:file "handler")))
Key :serial t indicates that asdf has uploaded files in the same order in which they are listed.
Now we need to write the file load.the lisp that will load our package and start the swank server for slime.
the
(in-package :cl-user)
(quicklisp:quickload "swank")
(quicklisp:quickload "usocket")
(quicklisp:quickload "bordeaux-threads")
(quicklisp:quickload "trivial-utf-8")
(quicklisp:quickload "cl-log")
(quicklisp:quickload "local-time")
(pushnew '*default-pathname-defaults* asdf:*central-registry*)
(asdf:load-system 'myweb)
(swank:create-server)
To continue development, we need to start swank and load all required libraries using quicklisp. To do this, run sbcl, being in the myweb directory, and call the function (quicklisp:quickload "swank"). After installing swank launch a swank server, by calling (swank:create-server) from the command line sbcl.
Let's start with the web server. For him, we need sockets. To work with sockets, I decided to implement with greatly expanded performance class libraries usocket. Also, we need threads (threads), for which we will use bordeaux-threads. But first I would like to tell you about the model of processing http requests we are going to create. Each request is processed by a separate thread. We will have threads-worker s to be created depending on the number of requests. Among them we will have a separate idle-threads that after completion of the processing of the request will go to the state of condition-wait, waiting for new requests. It is thus possible to reduce the load from creating new worker threads. It is a kind of mechanism thread pool for processing http requests.
Let's start with the announcement of socket and mutex variables for s in web.lisp:
the
(defvar *listen-socket* nil)
(defvar *listen-thread* nil)
(defvar *request-mutex* (make-lock "request-mutex"))
(defvar *request-threads* (list))
(defvar *worker-mutex* (make-lock "worker-mutex"))
(defvar *workers* (list))
(defvar *worker-num* 0)
(defvar *idle-workers* (list))
(defvar *idle-workers-num* 0)
(defvar *request-queue* (list))
To accept and distribute requests for flows we will use a separate thread, a pointer to which is stored in the *listen-thread*. Let's start with the start method-http:
the
(defun start-http (host port &key (worker-limit 10) (idle-workers 1))
(if (not *listen-socket*)
(setq *listen-thread*
(make-thread (lambda () (http-acceptor host port worker-limit idle-workers)) :name "socket-acceptor"))
"http server already started"))
It's a simple function to start the thread-dispenser, which in turn will call the function http-acceptor. We also have two key workers-limit — maximum number of worker s, and idle-workers — number of idle worker-ov.
Write the actual function of the distribution of the queries:
the
(defun http-acceptor (host port worker-limit idle-workers)
(setq *listen-socket* (socket-listen host port :reuse-address t :element-type '(unsigned-byte 8) :backlog (* worker-limit 2)))
(let ((request-id 0)
(worker-id 0))
(loop while *listen-thread* do
(let* ((socket (socket-accept *listen-socket* :element-type '(unsigned-byte 8))))
(progn (setq request-id (1+ request-id))
(acquire-lock *worker-mutex*)
(if (>= *worker-num* worker-limit)
(push (cons request-id socket) *request-queue*)
;; Get worker from idle workers
(if (> *idle-workers-num* 0)
(progn (push (cons request-id socket) *request-queue*)
(condition-notify (caar *idle-workers*)))
;; Add new Worker
(progn (setq worker-id (1+ worker-id))
(setq *worker-num* (1+ *worker-num*))
(setq *workers* (cons (make-thread (lambda () (worker-thread request-id socket idle-workers))
:name (concatenate 'string "socket-worker-" (prin1-to-string worker-id))) *workers*)))))
(release-lock *worker-mutex*)
t)))))
The first thing we do is socket-listen on the specified address and port. Further in a cycle we do the socket-accept, resulting in socket on the connected client that we have to process to the worker. Plus, we next request request-id. At this stage we must decide what to do with the request and how to handle it. First, we check the number of idle threads. If we have all the worker's are busy, we add the request to the queue for processing. If we have a free idle worker, then we again add the request in a queue, but this time caused (condition-notify (caar *idle-workers*))). And in the third case, we simply create new worker and send him a request that will be processed in the function worker-thread. Everything is quite simple. Only need to write the function processing of the worker thread:
the
(defun worker-thread (request-id socket idle-workers)
(if request-id
;; Process the request if it is not nil
(progn
(with-lock-held (*request-mutex*)
(setq *request-threads* (cons (cons request-id (current-thread)) *request-threads*))
)
(http-worker socket)
(with-lock-held (*request-mutex*)
(setq *request-threads* (remove-if (lambda (x) (eq (car x) request-id)) *request-threads*))
)
))
(acquire-lock *worker-mutex*)
(if *request-queue*
(let ((request nil))
(setq request (car *request-queue*))
(setq *request-queue* (cdr *request-queue*))
(release-lock *worker-mutex*)
(worker-thread (car request) (cdr request) idle-workers))
(if (< *idle-workers-num* idle-workers)
(let ((condition (make-condition-variable))
(idle-lock (make-lock))
(request nil))
(push (cons condition (current-thread)) *idle-workers*)
(setq *idle-workers-num* (1+ *idle-workers-num*))
(release-lock *worker-mutex*)
(list-workers)
(with-lock-held (idle-lock)
(condition-wait condition idle-lock)
)
(with-lock-held (*worker-mutex*)
(setq *idle-workers* (cdr *idle-workers*))
(setq *idle-workers-num* (1- *idle-workers-num*))
(setq request (car *request-queue*))
(setq *request-queue* (cdr *request-queue*))
)
(worker-thread (car request) (cdr request) idle-workers))
(progn (setq *workers* (remove (current-thread) *workers*))
(release-lock *worker-mutex*)))))
If we had a call with a request-id, we need to first handle the request. We simply call the auxiliary function http-worker and pass it the client socket. Next we check that there is still processing requests: just remove the first request from the queue and pass it to worker thread for processing, thereby the function of the worker-thread recursively. The question may arise "and whether there will be recursion limit from the fact that the stack overflows at some point, for example, for a large number of requests in the queue?" So after calling a worker-thread recursively we did in the function is not called, the recursion limit is not going to happen. Almost all contemporary Common Lisp implementation supports this optimization. Well, if the queue is empty, then we have to check the number of idle worker-ov. If we have everything in order, we simply complete the request and remove the worker from the list worker-ov. If not, then we make the condition-wait, and thus the worker becomes idle worker.
If you noticed, we also call the list-workers. This helper function, which simply clears the worksheet worker s from dead threads.
Left to write http-worker function:
the
(defun http-worker (socket)
(let* ((stream (socket-stream socket))
(request (myweb.util:parse-request stream)))
(myweb.handler:process-request request stream)
(finish-output stream)
(socket-close socket)))
(defun list-workers ()
(with-lock-held (*worker-mutex*)
(setq *workers*
(remove-if (lambda (w) (not (thread-alive-p w))) *workers*))
(setq *worker-num* (length *workers*))
*workers*))
Here we create a socket-stream, parsim request and pass it as myweb.handler:process-request (about these functions we will discuss in the second part). list-workers simply returns us the list worker-ov, pre-clearing it from dead threads. We call this function in worker-thread before condition-wait.
The last thing we need to do is to write a stop function-http, which will stop our web server:
the
(defun stop-http ()
(if *listen-socket*
(progn (stop-thread)
(socket-close *listen-socket*)
(setq *listen-socket* nil)
(setq *request-queue* nil)
(setq *worker-num* 0)
(setq *workers* nil)
(mapcar (lambda (i) (destroy-thread (cdr i))) *idle-workers*)
(setq *idle-workers-num* 0)
(setq *idle-workers* nil)
(release-lock *worker-mutex*)
(setq *request-threads* nil)
(release-lock *request-mutex*)
(setq *request-mutex* (make-lock "request-mutex"))
(setq *worker-mutex* (make-lock "worker-mutex")))))
(defun stop-thread ()
(if (and *listen-thread* (thread-alive-p *listen-thread*))
(destroy-thread *listen-thread*)))
As you can see, it's simple — we stop the flow dispenser, kill all worker-s and reset the list.
And so, all ready to handle our requests. We'll talk about that in second part.
Thank you for your attention!
PS Thanks ertaquo for help with spelling and layout-ohms
Комментарии
Отправить комментарий