Sunday, 28 January 2018

Creating web application with Spark Java framework


Spark is  a Java framework that let's you create web application. In this post we will see how we can write a basic web application using Java Spark framework.  Do not confuse this with Apache Spark which is a big data framework.  If you want to quickly bring up a local server to test something out Spark Java let's you do it in the simplest way possible. You do not need application server. It embeds Jetty server inside it.


Add following dependencies in your pom.xml for gradle build.


spark-core is the spark framework whereas slf4j-simple is for logging. Once above setup is done we can proceed to actual implement our rest application.

Getting Started with Java Spark

Following is a simple Spark code that starts a server and returns "Hello World!" in the response -

import spark.Request;
import spark.Response;
import spark.Route;
import spark.Spark;

 * @author athakur
public class HelloWorldWithSpark {

    private static final Logger logger = LoggerFactory.getLogger(HelloWorldWithSpark.class);

    public static void main(String args[]) {
        Spark.get("/", new Route() {

            public Object handle(Request request, Response response) throws Exception {
                logger.debug("Received request!");
                return "Hello World!";


Just run above Java code. It should start a jetty server and start listening for incoming requests. Default port that server listens on is 4567. So after running above code go to the browser and access following url  -
You should see "Hello World!" in the response.

Spark exposes static methods that let you define the URLs or routes you want to do some processing on and return some response. In above example we are listening on path "/" which is the root path and returning "Hello World!".

Same code in Java 8 perspective using functional programming/lambda would be -

public class HelloWorldWithSpark {

    private static final Logger logger = LoggerFactory.getLogger(HelloWorldWithSpark.class);

    public static void main(String args[]) {
        Spark.get("/", (req,res) -> {
            return "Hello World!";


NOTE : Here we are using GET verb but you can use any like POST, PUT etc.

You can easily create REST APIs from this. Sample example given below -

public class HelloWorldWithSpark {

    private static final Logger logger = LoggerFactory.getLogger(HelloWorldWithSpark.class);

    public static void main(String args[]) {
        Spark.get("/employee/:id", (req,res) -> {
            logger.debug("Got request to get employee with id : {}", req.params(":id"));
            return "Retrieved Employee No " + req.params(":id");
       "/employee/:id", (req,res) -> {
            logger.debug("Got request to add employee with id : {}", req.params(":id"));
            return "Added Employee No " + req.params(":id");


That was simple. Wasn't it? You want to deploy it in production like an actual web application in form of war you need to follow a bit different steps -

Related Links

Saturday, 20 January 2018

Difference between a forward proxy and a reverse proxy server


Most of the companies out there have a proxy in between their corporate traffic and internet. This could be for multiple reasons - network security being one of them. In my previous post I showed how to set up a squid proxy -
That was basically a forward proxy. There are other type of proxies called reverse proxies. In this post we will see difference between them and how they work.

Proxy in lay man terms mean someone acting on behalf of someone else. This is the main principle behind forward and reverse proxy.

Forward Proxy :

Working :

Forward proxy sits between client machines and an origin server. Client machines make a request to the forward proxy with target as the origin server. Forward proxy then makes a request to the origin server, gets the response and sends it back to the clients. Clients in this case need to be explicitly configured to use this kind of forward proxy.

So to summarize a forward proxy retrieves data from another website (origin server) on behalf of the clients.

Example : 

 Consider three computers - A, B and C. Now A want to request a website hosted on computer C. In normal case it would directly be
  • A -> C
where computer A directly asks C for the website. However in case of Forward proxy there is an intermediate computer B. Computer A makes request to this computer B instead of directly making request to C. Computer B now makes a request to C gets the website and returns it back to the A. So the path would be
  • A -> B -> C.

When :

There can be multiple cases in which a forward proxy might be useful. Some are -
  • Client machines (Computer A in above case) are behind some firewall and have no access to internet and thereby no access to origin server.
  • A company wants to block some of the malicious sites. They do this on the forward proxy and make sure all client make request via this proxy.
  • A forward proxy also has feature to cache requests so that the response time is minimum.

Reverse Proxy :

Working : 

Forward proxy was used to shield the client machines where as a reverse proxy is used to shield a origin server. So client machines make call to the reverse proxy as if they are the origin servers. Reverse proxy now makes a call to the actual origin server and returns the response back to the client.

Example :
Let's consider a similar example of 3 computers - A,B and C. Again in a normal scenario A would directly request website from C.
  • A -> C
In case of reverse proxy there is a computer B which hides C behind it. A makes call to B instead and B fetches the website from C and returns it back to A. So the path is again -
  •  A -> B -> C
  • Provide internet users access to a server that is behind the firewall.
  • Load balance backend servers.
  • Typical CDN deployment. Proxy server would tell the client the nearest CDN server.

 Difference between a proxy and a reverse proxy server

 If you see the example above in case of both forward and reverse proxy path is always -
  • A -> B -> C
In case of forward proxy B shields machine A by fetching content by C itself and sending back to A. Where as in case of reverse proxy B shields C by fetching the data from C and sending it back to A.

In case of forward proxy, C would think that B is the machine sending it request where there could be multiple A's behind B. Similarly  in case of reverse proxy A would think that it is sending request to C but it would actually be a request to B and B would in turn talk to multiple C and send back the request to A.

Related Links

Thursday, 18 January 2018

How to set up a squid Proxy with basic username and password authentication in Ubuntu


Most of the big companies have their own proxies through which all the company data is routed through. This ensure malicious sites are blocked and all other traffic is audited via proper authentication. 

To give a little background on Squid proxy -
Squid is a caching and forwarding HTTP web proxy. It has a wide variety of uses, including speeding up a web server by caching repeated requests, caching web, DNS and other computer network lookups for a group of people sharing network resources, and aiding security by filtering traffic. Although primarily used for HTTP and FTP, Squid includes limited support for several other protocols including Internet Gopher, SSL,[6] TLS and HTTPS. Squid does not support the SOCKS protocol.

Squid was originally designed to run as a daemon on Unix-like systems. A Windows port was maintained up to version 2.7. New versions available on Windows use the Cygwin environment.[7] Squid is free software released under the GNU General Public License.

Source : Wiki

Installing Squid proxy on Ubuntu

To install squid server simply run following command in your terminal -
  • sudo apt install squid

Squid run as daemon service in Ubuntu. You can execute following command to see the status of this service -
  • service squid status
It will show you if squid service is running or not.

Some important file paths are -
  • /etc/sqid :  This is where your squid configuration resides
  • /var/log/squid : This is where your squid logs reside
  • /usr/lib/squid3,/usr/lib/squid : This is where your squid modules or libraries reside.
Now that we have Squid proxy installed. Let's configure it.

Squid configuration is located at -
  • /etc/squid/squid.conf
Before you make changed to this file make a copy of this and store it aside. Use following commands to do that -

  • sudo cp /etc/squid/squid.conf /etc/squid/squid.conf.original
  • sudo chmod a-w /etc/squid/squid.conf.original 
This essentially created a copy of  squid.conf called squid.conf.original and removed all write access to it so that no one can accidentally write it.

Default TCP port that Squid listens to is 3128. Go ahead and change it to 8888. I prefer using 8888 port since this is used by other proxies as well like Charles and Fiddler. To do this find a line called

  • http_port 3128
and change it to

  • http_port 8888

Next you need to provide rules to allow and disallow traffic. If you want to just allow trafic from your local machine you can add the following lines to the configuration -
  • acl localhost src
  • http_access allow localhost 
acl is nothing but access control list. it's a keyword that states acl is starting. Next localhost is the name that is used to indentify the acl. I have named it localhost but it can be anything. Next we have src which is used to identify local IP addresses. Other options are -
  1. srcdomain  : used for declaring local domain, 
  2. dst : used for public IP & 
  3. dstdomain : used for public domain name
Next  we have http_access that will basically take action provide in it's next word on the acl we define. In this we we are saying allow and for acl named localhost that we defined above. So Squid proxy is going to allow all http traffic from local machine (i.e with IP

Last line you can add as  -
  • http_access deny all
which says you deny all other traffic. So the way acl's work is -

For each request that Squid receives it will look through all the http_access statements in order until it finds a line that matches. It then either accepts or denys depending on your setting. The remaining rules are ignored. 

This was basic settings for squid proxy. Now let's see how we can add an authentication to this scheme.

Post configuration you can just restart the squid service -
  • service squid restart
You can also view the service logs for this in file-
  • less /var/log/squid/cache.log
 And you can view the access logs in file -

  • less /var/log/squid/access.log

How to set up a squid Proxy with basic username and password authentication?

For this you can add following lines to your squid configuration file squid.conf -

auth_param basic program /usr/lib/squid3/basic_ncsa_auth /etc/squid/passwords
auth_param basic realm proxy
acl authenticated proxy_auth REQUIRED
http_access allow authenticated

ident_lookup_access deny all
http_access deny all

Above configuration will ensure all traffic is authenticated. The username/password that would be needed to provide access will be stored in a file - /etc/squid/passwords. We will now see how we can create this file.

To generate username/passwrod you need to use a command called htpasswd. You can install this using -
  • apt-get install apache2-utils
Next to generate username/password type in following command -
  • sudo htpasswd -c /etc/squid/passwords YOUR_USERNAME
Replace  YOUR_USERNAME with the user name you want. Eg admin. You will be prompted for password for this username twice. Once done your user is all setup. You can use this credentials to access your proxy.

NOTE : htpasswd stores the password hashed.

One done you can restart your squid service -
  • service squid restart
My conf file looks like below -

acl SSL_ports port 443
acl Safe_ports port 80          # http
acl Safe_ports port 21          # ftp
acl Safe_ports port 443         # https
acl Safe_ports port 70          # gopher
acl Safe_ports port 210         # wais
acl Safe_ports port 1025-65535  # unregistered ports
acl Safe_ports port 280         # http-mgmt
acl Safe_ports port 488         # gss-http
acl Safe_ports port 591         # filemaker
acl Safe_ports port 777         # multiling http

http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports

http_port 8888

coredump_dir /var/spool/squid

refresh_pattern ^ftp:           1440    20%     10080
refresh_pattern ^gopher:        1440    0%      1440
refresh_pattern -i (/cgi-bin/|\?) 0     0%      0
refresh_pattern (Release|Packages(.gz)*)$      0       20%     2880
refresh_pattern .               0       20%     4320

auth_param basic program /usr/lib/squid3/basic_ncsa_auth /etc/squid/passwords
auth_param basic realm proxy
acl authenticated proxy_auth REQUIRED
http_access allow authenticated

ident_lookup_access deny all
http_access deny all 

Now you can test this by adding a proxy in firefox and trying to go to a http URL.

Add username/password that you just created before and the URL should be accessible.

Related Links

Saturday, 6 January 2018

Writing your first Django app - PART 1


Django is a python based web framework that let's you create webapps quickly and with less code. It's free and opensource. For more details on the framework itself visit -
In this post we will create a sample app in Django python framework.


This tutorial assumes you are using Django 2.0, which supports Python 3.4 and later.

Install python3 and pip3 -
  • sudo apt-get install python3-pip
Next install Django python framework using pip3 -
  • sudo pip3 install Django 
You can see the installed version of python and django in various ways. Some are given in screenshot below -

Creating a Django project

Create a skeleton of your Django app using following command -
  • django-admin startproject djangodemo
You should see a directory getting created with name djangodemo. Inside this you should have file and another directory with same name djangodemo. This inner directory named djangodemo is actually a python package. Outer directory is just a holder with file. file is used to give you command line tasks to interact with your django project. You can see the version of you django framework used with following command -
  •  python3 version 

Directory structure is as follows -

 Some other pointers other than ones mentioned above -
  • tells python this directory should be considered as a package.
  • This also means your inner djangodemo directory is a python package.
  • Your Django app settings go here.
  • URLs used in your Django project go here.
  • This is an entry-point for WSGI-compatible web servers that can serve your project.
 Now that you have created your project let's run it with following command -
  • python3 runserver

Ignore the warnings for now.

NOTE : You don't have to restart the server everytime you make changes to code. Django handles it. Just refresh the pages.

Open -
 You should see installation successful message as follows -

NOTE : By default your server will run on port 8000. But you can change it as follows -
  • python runserver 8080

Creating Django App

A project is collection of apps and it's configurations needed for a website to run. Apps are modules that run in your project. A project can have multiple apps. Similarly a app can be part of multiuple projects. Apps can be at any python paths.

You can create a app as follows -
  • python3 startapp testapp
I like to put all apps in a directory called apps in your actual python package directory. You can do that as follows -

Creating your webpage

Go to your apps directory and edit to add following content -

from django.http import HttpResponse

def index(request):
    return HttpResponse("Hello world!")

 Next in the same directory create a file called and add following content to it -

from django.urls import path
from . import views

urlpatterns = [
    path('', views.index, name='index'),

Finally go to your project directory  - djangodemo/djangodemo and edit file to have following content -

from django.contrib import admin
from django.urls import path, include

urlpatterns = [
    path('test/', include('djangodemo.apps.testapp.urls')),

Next in apps directory inside djangodemo directory create a file called You can do this using -
  • touch
Now simply run your server and visit -
to see your site.

Understanding : First we created an app called testapp. It should have some default files like stores all your views. Here we added a new view called index and mapped it inside a file to the root url ("") at the app level. Next we mapped this to at our project level for '/test'. include maps the url provided and forwards rest the included module. In this case it will check url has 'test/' and forward the rest which is - "" to the in the testapp where we have mapped request view to "". So request view gets rendered.

NOTE : Note how we added a file in apps directory. This is to ensure python recognizes this directory as a package. So that we could use djangodemo.apps.testapp.urls in the of project.

That's it you created your 1st django project and app. We will see some more details about this in next post. Thanks.

Related Links

t> UA-39527780-1 back to top