Wednesday, 12 April 2017

Using HttpUrlConnection in Android


Android has two options for doing network operations using Http clients -
  1. Apache HttpClient
    1. DefaultHttpClient
    2. AndroidHttpClient
  2. HttpUrlConnection

Back in 2011 Android developers announced that Android team is not actively working on Apache HTTP Client and that  new android applications should use HttpURLConnection and that is where they will be spending their energy going forward.

For more details you can read their blog -  Android’s HTTP Clients 

It's been a long way since they has posted that post and here we are. Android Http client was totally removed from Android 6.0 (M). Ofcourse you can manually plug it in. But android developer site says the following -

Android 6.0 release removes support for the Apache HTTP client. If your app is using this client and targets Android 2.3 (API level 9) or higher, use the HttpURLConnection class instead. This API is more efficient because it reduces network use through transparent compression and response caching, and minimizes power consumption. To continue using the Apache HTTP APIs, you must first declare the following compile-time dependency in your build.gradle file:

android {
    useLibrary 'org.apache.http.legacy'

But I would recommend don't do this unless some legacy API needs this. Move on to HttpUrlConenction. In this post we will see how we can use HttpUrlConnection for network operations in android.

Using HttpUrlConnection in Android

HttpURLConnection is a general-purpose, lightweight HTTP client suitable for most applications. A simple snippet would be- 

        URL url = new URL("");
        HttpURLConnection urlConnection = (HttpURLConnection) url.openConnection();
        try {
            InputStream in = new BufferedInputStream(urlConnection.getInputStream());
        } finally {

So you open a connection by calling openConnection() on an URL instance. You can then set the type of request using setRequestMethod(). It can be GET, POST etc. Then you read the stream by calling urlConnection.getInputStream(). Finally you call disconnect() to release your connection.

NOTE : By default HttpUrlConnection will use GET as request type i.e you dont have to mention  urlConnection.setRequestMethod("GET"); . For others as started above you can use - setRequestMethod();

NOTE :  Connection is actually established when you call getInputStream() or getOutputStream() or getResponseCode() on your httpUrlConenction instance.

NOTE : If your response is not 2** then it would be an error(You can read the response code by calling getResponseCode() method). In that case you need tor read error stream. You can do that by using getErrorStream() to read the error response. The headers can be read in the normal way using getHeaderFields().

How disconnect() works?

    Each HttpURLConnection instance is used to make a single request but the underlying network connection to the HTTP server may be transparently shared by other instances. Calling the close() methods on the InputStream or OutputStream of an HttpURLConnection after a request may free network resources associated with this instance but has no effect on any shared persistent connection. Calling the disconnect() method may close the underlying socket if a persistent connection is otherwise idle at that time.   

 -> So basically your HttpUrlConnection instance may get GCed but the underlying TCP socket will get pooled and reused unless you call disconnect() on it in which case it may or may not close the socket. So if you are not reconnecting to same URL it is safe to always call disconnect and let android handle the socket pooling and closing part.

Android docs clearly states -  Disconnecting releases the resources held by a connection so they may be closed or reused.

Handling request body

Sometime a request made to a server may need a request body to be supplied. For that you can use the httpUrlConnection instance's outputStream.

        URL url = new URL("");
        HttpURLConnection urlConnection = (HttpURLConnection) url.openConnection();
        try {
            OutputStream out = new BufferedOutputStream(urlConnection.getOutputStream());

            InputStream in = new BufferedInputStream(urlConnection.getInputStream());
        } finally {

Make sure you set setDoOutput(true) if you have request body to be sent. Similarly set setDoInput(true) if you are reading back from the connenction. If you just want to see response code using - getResponseCode() method that these are not required.

NOTE : Notice how we have use BufferedStreams over input/output streams. They are used for performance reasons - so that data is buffered and we can easily read/write it.

Setting headers and connection timeouts

If you are familiar with HttpClient then you must have set connection and socket timeouts. In HttpUrlConenction you need to do following -


These are in milliseconds. I have set it to half a minute above. To set headers in the request you can do -

urlConnection.setRequestProperty("headerName", "headerValue");

So lets say you want to set user-agent in your request you do -

urlConnection.setRequestProperty("User-Agent", "Mozilla");

Similarly you can enable disable redirects using -

Making secure connection (https)

If you are working with https (secure connection) protocol then your actual class would be - HttpsUrlConnection.

NOTE :  You can still use HttpUrlConnection since HttpUrlConenction extends HttpsURLConnection (by polymorphism).

HttpsUrlConnection will allow you to override HostnameVerifier and SSLSocketFactory.


static {
    TrustManager[] trustAllCertificates = new TrustManager[] {
        new X509TrustManager() {
            public X509Certificate[] getAcceptedIssuers() {
                return null; // Not relevant.
            public void checkClientTrusted(X509Certificate[] certs, String authType) {
                // Do nothing. Just allow them all.
            public void checkServerTrusted(X509Certificate[] certs, String authType) {
                // Do nothing. Just allow them all.

    HostnameVerifier trustAllHostnames = new HostnameVerifier() {
        public boolean verify(String hostname, SSLSession session) {
            return true; // Just allow them all.

    try {
        System.setProperty("jsse.enableSNIExtension", "false");
        SSLContext sc = SSLContext.getInstance("SSL");
        sc.init(null, trustAllCertificates, new SecureRandom());
    catch (GeneralSecurityException e) {
        throw new ExceptionInInitializerError(e);

Sample take from this SO discussion.

NOTE :   Calling HttpsURLConnection.setDefaultSSLSocketFactory() or HttpsURLConnection.setDefaultHostnameVerifier() will set it up for all Https connections. If you want to set it for specific connections then type cast it to HttpsUrlConenction instance and then call set methods on it.

Handling multipart/ form-data

 You'd normally use multipart/form-data encoding for mixed POST content (binary and character data). The encoding is in more detail described in RFC2388.

String param = "value";
File textFile = new File("/path/to/file.txt");
File binaryFile = new File("/path/to/file.bin");
String boundary = Long.toHexString(System.currentTimeMillis()); // Just generate some unique random value.
String CRLF = "\r\n"; // Line separator required by multipart/form-data.
URLConnection connection = new URL(url).openConnection();
connection.setRequestProperty("Content-Type", "multipart/form-data; boundary=" + boundary);

try (
    OutputStream output = connection.getOutputStream();
    PrintWriter writer = new PrintWriter(new OutputStreamWriter(output, charset), true);
) {
    // Send normal param.
    writer.append("--" + boundary).append(CRLF);
    writer.append("Content-Disposition: form-data; name=\"param\"").append(CRLF);
    writer.append("Content-Type: text/plain; charset=" + charset).append(CRLF);

    // Send text file.
    writer.append("--" + boundary).append(CRLF);
    writer.append("Content-Disposition: form-data; name=\"textFile\"; filename=\"" + textFile.getName() + "\"").append(CRLF);
    writer.append("Content-Type: text/plain; charset=" + charset).append(CRLF); // Text file itself must be saved in this charset!
    Files.copy(textFile.toPath(), output);
    output.flush(); // Important before continuing with writer!
    writer.append(CRLF).flush(); // CRLF is important! It indicates end of boundary.

    // Send binary file.
    writer.append("--" + boundary).append(CRLF);
    writer.append("Content-Disposition: form-data; name=\"binaryFile\"; filename=\"" + binaryFile.getName() + "\"").append(CRLF);
    writer.append("Content-Type: " + URLConnection.guessContentTypeFromName(binaryFile.getName())).append(CRLF);
    writer.append("Content-Transfer-Encoding: binary").append(CRLF);
    Files.copy(binaryFile.toPath(), output);
    output.flush(); // Important before continuing with writer!
    writer.append(CRLF).flush(); // CRLF is important! It indicates end of boundary.

    // End of multipart/form-data.
    writer.append("--" + boundary + "--").append(CRLF).flush();

Again code is taken from this SO discussion.

Other advantages of using HttpUrlConnection

  • HttpUrlConnection by default honors system proxy. However if you want to specify a custom proxy then you can do so with API - url.openConnection(proxy)
  • It also has transparent response compression. It will automatically add the header - Accept-Encoding: gzip to outgoing responses and also handle it automatically for incoming connections.
  •  It also attempts to connect with Server Name Indication (SNI) which allows multiple HTTPS hosts to share an IP address.

Related Links

Sunday, 2 April 2017

Android material design floating labels on EditText and Password visibility Toggle


With android design support library new element - TextInputLayout was introduced. With this we can totally omit static labels that we put before text input elements (EditText for example). With this label will float on the top of the EditText when user focuses on that field for entering data. Also we can show error on the same field post validation. That gets show floating below the EditText. It also has passwordToggleEnabled   which lets you toggle the visibility of your password field with an eye icon (it can be customized to use a different icon too). We will see all of these in this post.

Android material design floating labels on EditText and Password visibility Toggle

Make sure your app level gradle has following dependencies added -

    compile ''
    compile ''

NOTE : Base version of you appcompat and your design library should be same. For eg. it's 25 for this case.

Next lets go straight to the layout. The feature that we are looking for goes something like this -




Now I have combined both floating labels and password toggle in a single example. Lets see how it works now.

  • First of all you need to wrap  EditText element inside TextInputLayout element. 
  • Next android:hint value of EditText will come out on top as a floating label.
  • For password visibility to be toggle we need to provide app:passwordToggleEnabled="true" attribute. Others are optional really. Lets see what all attributes it supports.

  1.  passwordToggleEnabled : allows you choose whether or not you want the password visibility to be toggled.
  2. passwordToggleContentDescription :  allows a string to be set for the content description. This is used for accessibility feature.
  3. passwordToggleDrawable : allows one set a different icon other than the default visibility toggle icon (eye icon). 
  4. passwordToggleTint : allows the visibility toggle icon to be tinted to whatever colors you indicate.
  5. passwordToggleTintMode : sets the blending mode used to apply the background tint.
You can see the complete specs in android developers site. It would look like below -

Final screen is just showing a toast when you press login and all validations have gone though successfully.Speaking of validation lets come to it.

Do not worry about the minute code details like string resources. You can find the complete code on my github repo -
Lets just focus on concepts for now.

We said in our initial discussion that TextInputLayout can handle errors as well and they are shown floating just below the EditText. Lets look at the code for it.

  1. In your activity's onCreate() method get a reference to the TextInputLayout element and reference to the EditText wrapped in it.
  2. Next you can addTextChangedListener() to your EditText and do validations there.
  3. To set and display an error you simply need to call setError()on textInputLayout element. To remove error you can simply call setErrorEnabled(false).
Lets see how we can do it in case of our Email field. First of all get a reference to TextInputLayout and the EditeText in it -

        inputLayoutEmail = (TextInputLayout) findViewById(;

        inputEmail = (EditText) findViewById(;

Next add addTextChangedListener to inputEmail and in it's afterTextChanged() method simply do your validations. In this case we are just calling a private validate method from afterTextChanged() method as follows -

    private boolean validateEmail(String text) {
        if(!isValidEmail(text)) {
            return false;
        else {
        return true;

    private boolean isValidEmail(String email) {
        return !TextUtils.isEmpty(email) && android.util.Patterns.EMAIL_ADDRESS.matcher(email).matches();

and you are done. You should see error as shown in following screenshots -

Again for complete code please refer to my github repo -

Related Links

Saturday, 1 April 2017

Android : URI is not registered (Settings | Project Settings | Schemas and DTDs)


When you are developing android application you might sometime see following error on your layouts xml files in Android studio-
  • URI is not registered (Settings | Project Settings | Schemas and DTDs)

Everything seems correct except this error is persistent. Gradle sync works fine too. Lets see how we can resolve this.


  • Before we see how to resolve this issue make your layout file is in correct directory structure. It should be under layout folder which would be under res folder.

  • Now look at build variant tab at the bottom left of android Studio. You should see the current version set. It could be -
    • debug OR
    • release
In my case it was release. Just switch the version and then switch it back. In my case I switched it to release and then back to debug and the error was gone.

  •  If above 2 methods did not work out for you then Close all open tabs and reopen them. 
  • Last but not the least - restart Android Studio.
I know these are just silly workaround. Let me know which one works for you :)


Wednesday, 29 March 2017

CS 1.6 redirection to other server issue


This is not a technical post rather a solution for the issue I faced while playing CS 1.6 online. I had bought it on Steam and happen to play around online on different servers. After a while it seemed to be redirecting to other servers and that seems to be issue. How can it connect to a server without Me telling it to? Well well see that in a moment.

I have steam installed and playing CS 1.6 on my mac. So path and files location may be different for you.

CS 1.6 redirection to other server issue

First of all find your counter strike installation directory. For me it is -
  • /Users/athakur/Library/Application Support/Steam/steamapps/common/Half-Life/cstrike
It would be different for you based on which OS you are using and how did you install it.

One you locate it try to find files with .cfg extension. The file we are interested in is 
  • config.cfg OR
  • userconfig.cfg  (If you have it)
In these files look for bind commands that ask to redirect to some IP. For eg. -
  • bind "F12" "Connect";
When you connect to a rogue server it can alter these files to change your bindings and user configuration. You can see the permissions of these files. Your user would have write permissions on  it.

NOTE : Word of advice. Do not connect to servers that don't have folks playing in it and the ones you don't trust.

Now that we know what the issue is lets see how we can fix it.

Solution : CS 1.6 redirection to other server issue

  1. Close your CS game and delete the compromised config file.
  2. Restart you game and you can see that the config.cfg file is created again with default values. You can also go in game and click on reset to defaults. Do not connect to any servers.
  3. Once you have confirmed your file exist and you have made chances to it based on your configurations (binding keys etc) you need to make it read only.
  4. For this you can execute chmod ugo-w config.cfg command i.e you remove write permissions for the file. And your gamming life will be back to normal :)

NOTE : For me it was just config.cfg but if you open config.cfg it internally execute userconfig.cfg. So make sure if you have it, it is not compromised too.

 Related Links

Sunday, 19 March 2017

Creating a new Xposed module in Android

How Xposed works

Before we start lets see how Xposed works -

There is a process that is called "Zygote". This is the heart of the Android runtime. Every application is started as a copy ("fork") of it. This process is started by an /init.rc script when the phone is booted. The process start is done with /system/bin/app_process, which loads the needed classes and invokes the initialization methods.

This is where Xposed comes into play. When you install the framework, an extended app_process executable is copied to /system/bin. This extended startup process adds an additional jar to the classpath and calls methods from there at certain places. For instance, just after the VM has been created, even before the main method of Zygote has been called. And inside that method, we are part of Zygote and can act in its context.

The jar is located at /data/data/ and its source code can be found here. Looking at the class XposedBridge, you can see the main method. This is what I wrote about above, this gets called in the very beginning of the process. Some initializations are done there and also the modules are loaded.

Creating a new Xposed module in Android

I have couple of app on playstore . I am going to use FlashLight to demonstrate Xposed module.

  • Create a normal Android app. Add following extra meta data in the apps manifest file. This is how xposed installer app knows your apk is a xposed module.

            android:value="true" />
            android:value="Demo example that renders Flashlight app (com.osfg.flashlight) useless" />
            android:value="53" />

  • Next create a class that implements IXposedHookLoadPackage like below -

public class XPFlashLightKiller implements IXposedHookLoadPackage {
    public void handleLoadPackage(XC_LoadPackage.LoadPackageParam loadPackageParam) throws Throwable {

        if (!loadPackageParam.packageName.equals("com.osfg.flashlight"))

        XposedBridge.log("Loaded app: " + loadPackageParam.packageName);

        XposedHelpers.findAndHookMethod("com.osfg.flashlight.FlashLightActivity", loadPackageParam.classLoader, "isFlashSupported", new XC_MethodHook() {
            protected void beforeHookedMethod(MethodHookParam param) throws Throwable {
                // this will be called before the clock was updated by the original method
            protected void afterHookedMethod(MethodHookParam param) throws Throwable {
                // this will be called after the clock was updated by the original method
                XposedBridge.log("Hooken method isFlashSupported of class com.osfg.flashlight.FlashLightActivity");

NOTE :  Here we have used XC_MethodHook as callback so that we can execute methods before and after original method is executed. Other alternative is to replace the original method entirely - original hooked method will never get executed.

When you call XposedHelpers.findAndHookMethod the callback can either be
  • XC_MethodHook : Callback class for method hooks. Usually, anonymous subclasses of this class are created which override beforeHookedMethod(XC_MethodHook.MethodHookParam) and/or afterHookedMethod(XC_MethodHook.MethodHookParam).
  • XC_MethodReplacement : A special case of XC_MethodHook which completely replaces the original method.

1st one just provides you the hooks to execute methods before and after original method where as 2nd one replaces it completely.

  • Next create a file called xposed_init in your assets folder and add your fully qualified class name to it. This file tells xposed installer where to look for the module class. For eg in our case it will be -

  • Finally download XposedbridgeApi.jar from here and add it in your app folder. Make sure the version of this jar should be same as xposedminversion set in the meta data tags in step 1.
Now build your app and deploy on your device. Once done Xposed installed should detect it. Just enable the module and reboot the device -

Testing out the module

Just to give you some background on the original Flashlight app. It's code is something like below -

    public void onRequestPermissionsResult(int requestCode, String[] permissions, int[] grantResults)
        switch (requestCode)
            case CAMERA_PERMISSION:
                if (grantResults.length > 0 && grantResults[0] == PackageManager.PERMISSION_GRANTED) {
                    Log.d(TAG, "Received Camera permission");
                    if(!isFlashSupported()) {
                        Toast.makeText(getApplicationContext(), "LED Flash is not supported on your device",Toast.LENGTH_LONG).show();

                else {
                    Log.d(TAG, "Camera permission denied");
                    Toast.makeText(this, "Please provide camera permission", Toast.LENGTH_SHORT).show();

It first asks for permission. If you give it then it will check if flash is supported on the device. So if you have already given Flashlight app camera permission then remove it from settings and open the app again. Now when it prompts for permissions again give it. But now you can see that the app shuts down with toast message - "Flash not supported on this device". This is because we hooked into isFlashSupported() method and made it always return false. So that the app never works :)

NOTE : From next time it will work fine. Since permission is already given next time it will not prompt and never execute  isFlashSupported() method. To retest remove the permissions from settings again.

You can also see following line in logcat -

03-20 02:36:37.864 8002-8002/? I/Xposed: Loaded app: com.osfg.flashlight

03-20 02:36:44.123 8002-8002/? I/Xposed: Hooken method isFlashSupported of class com.osfg.flashlight.FlashLightActivity

Complete code is added in my github repo -

Related Links

Installing Xposed Framework on Android devices


Android as you know is very flexible and developer friendly. It runs a Linux kernel underneath with some modifications needed to operate handheld devices with relatively less memory. You can root your device to further boost its potential application. You can do stuff with root access now. You can even take it a step further - Unlock boot-loader and install customer recovery like ClockworkMod or TWRP (We are going to work with TWRP in this post). Custom recovery software's let you install custom mods or even full ROMs that can drastically improve user experience.

Xposed is a framework that can alter the the way your applications work, give you more control over it without actually flashing any custom ROMs or MODs or even tampering any of the apks. It uses root privileges to access androids core resources and use it to change the androids behavior .It's potential power is unlimited. We will see how to install and use it in a while.

NOTE : Xposed uses root privileges i.e device needs to be rooted.

Xposed is a framework once installed you can create and install various modules to alter apps/android behavior. Best part about this is that the modules are held in memory. So to get back the system to original state all you need to do is uninstall the module and reboot the device.

NOTE : Works on Android 4.0.3 (ICS) and above only.

Xposed framework is brought to us by XDA Recognized Developer rovo89.

Warning : This required rooting your device and changing some androids base resources. Follow this guide on your own risk. We are not liable if you phone gets bricked or damaged in the process.

 Installation and Setup

Make sure your device is recognized by adb (connect your device to your laptop having android SDK using a USB) -

NOTE : If you adb is not able to detect or recognize your device then you may refer to link in the Related Section at the end of this post.

Before you start Xposed has two parts to it -
  1. Xposed Framework : Lays down all the groundwork for the framework
  2. Xposed Installer app : Apk use to manage Xposed modules.
Download the framework from -
Choose based on your the sdk version you have on your android device.

You can download the latest xposed installer from  -
First we are going to flash custom recovery image - TWRP on our Android device. So first reboot in bootloader mode using following command -
  • adb reboot bootloader
You device should reboot now and you should see following screen -

Once you reach above screen adb will not work. You need to use fastboot instead. You can type following command to make sure your device is still connected -
  • fastboot devices

 Now lets flash our recovery image. Use following commands to do so -
  • fastboot flash recovery /Users/athakur/Desktop/xposed/twrp-3.0.2-0-hammerhead.ndif
 You can download the image from here. Above command obviously has path to the image from my local machine. You need to replace it accordingly.

Once done navigate to Recovery mode by navigating using vol up/down button. Once on Recovery mode press power button to enter it.

Now go to install and install the Xposed framework - the zip file we downloaded earlier and then reboot.

On reboot install the Xposed installed apk. On install open it -

Here your Xposed  setup and installation is complete. You can create and install modules going in Modules section of installer as shown in screenshots above. We will see how to create modules in upcoming post. So stay tuned!

Good to Know 

What is Fastboot? : Fastboot is a protocol that can be used to re-flash partitions on your android device (update the flash file system in your Android devices). It comes with the Android SDK (Software Developer Kit) and can be user as an alternative to the Recovery Mode for doing installations and updates.

What is Android recovery image? : A bootable program on an Android flash memory partition (/recovery) that is used to perform a factory reset or restore the original OS version. You can also install a different OS version (a different ROM). For that the stock recovery image must be replaced with a custom version such as ClockworkMod Recovery or TWRP (as we are doing).

Related Links

Friday, 17 March 2017

Understanding HTTP Strict Transport Security (HSTS)


Have you any time tried to visit a site by typing "http://" in your web browser and the site that loads corresponds to "https://". Well it happens a lot of time. A website may support only https - secure connections and browser is smart enough to know this. How? We will see that in a moment.

My use case was for using a proxy to intercept browser traffic. As you know any proxy will intercept traffic and then send it's own certificate to the browser and if browser does not recognize the cert it shows the error -
  • "Your connection is not secure"... SEC_ERROR_UNKNOWN_ISSUER
You may go in Advanced tab and then add exception to start trusting the new certs for your testing. But it is not always possible and well see why?

NOTE : If you are not trying something on your own and dont understand why above prompt came its time to turn back. Your network traffic may be getting intercepted by some hacker using MITM (Man in the middle) attach to steal sensitive data like passwords.

WARNING : All the below information is provided only for the use case of pen testing and other security testing. Please do not use it otherwise. All of the below features are provided for your security. Playing around with it without understanding can compromise your security. So proceed on your own risk.

HTTP Strict Transport Security (HSTS)

As per Wiki,

HTTP Strict Transport Security (HSTS) is a web security policy mechanism which helps to protect websites against protocol downgrade attacks and cookie hijacking. It allows web servers to declare that web browsers (or other complying user agents) should only interact with it using secure HTTPS connections,[1] and never via the insecure HTTP protocol. HSTS is an IETF standards track protocol and is specified in RFC 6797.

How this works is that your webserver eg. Apache or NGINX is configured so that in each response header it sends header corresponding to  HSTS
  • Strict-Transport-Security "max-age=63072000"
max-age is the expiry time. So the HSTS will be applicable till this much time.

NOTE : This is updated every time you visit the site. So if this time period is 2 years them each time you visit the site this will be set for next 2 years.

For eg see below in case of techcrunch -

Working of HTTP Strict Transport Security (HSTS)

Browser once it receives stores this data corresponding to each site. In case of Firefox you can see it as follows -
  1. Type about:support in firefox tab
  2. Click on "Show in Folder". This should open your firefox profile folder.
  3. In this search for a file called SiteSecurityServiceState.txt and open it.
  4. Firefox stored HSTS data for sites in this file.
Steps in screenshots -

NOTE : As mentioned earlier this entry is updated every time you hit the url in your browser. Exception is incognito mode in which case entry from this file is removed once incognito is close. However note that if you have visited the site even once in non incognito mode then the security restrictions will be obeyed even in incognito mode.

Working :
  • This tells browser that every subsequent connection should always be https (secure). So even if you try to access http version of the site browser will convert it to https and proceed. 
    • For eg. You can try hitting "" and it will automatically hit ""
  • Another effect it has it that if this header is set then you cannot add exception for the certificate which does not match the original one (the one browser does not trust). Eg in screenshot below -

NOTE : I am using Zap proxy. You can use any others like Fiddler or Burp. I have written posts on this before you can revisit them if needed.

How to bypass HSTS?

Well now that we know where firefox stores this data it is easy to bypass this. All you have to do is remove the entry from this file. You can probably change the permission of this file so that new entry is not made in it. When tried for techcrunch you can see you can get back option to add certificate exception -

NOTE : For above to work make sure firefox is closed else it can overwrite the file.

NOTE : Above workaround will not work for the well known sites like google since for them the entries are hardcoded into browser code. Please do share if you know of any workaround for this.

NOTE : In the above mentioned file you might also see entries corresponding to HPKP. A HTTP Public Key Pinning (HPKP) header instructs clients to pin a specific public key to a domain. So, if a HPKP-supporting browser encounters a HPKP header, it will remember the specified public key hashes and associate them with that domain. In the future (until the specified max-age timeout expires), the browser will only accept a certificate for that domain if any key in the certificate's trust chain matches one of the associated hashes.

Wednesday, 8 March 2017

Understanding Cross-site scripting (XSS)


As per wiki
             Cross-site scripting (XSS) is a type of computer security vulnerability typically found in web applications. XSS enables attackers to inject client-side scripts into web pages viewed by other users. A cross-site scripting vulnerability may be used by attackers to bypass access controls such as the same-origin policy.
       We will see this in details as to how web applications can be vulnerable to an XSS attack and how can it be safeguarded. This is the most common known security vulnerability. You can use 3rd party produces like IBM App scan to check for security vulnerabilities in your web applications.

Types of XSS attacks

There are many variants of XSS attacks but the major ones are -
  • Reflected or Non persistent XSS attack
  • Stored or persistent XSS attack

Reflected or Non persistent XSS attack

Out of this reflected is most common. In this type if xss malicious script is executed on clients browser and sensitive data might be stolen.

For eg. lets take a search field on some social networking site. You generally type is words their to either search for a particular word in your feed or to search your friend/family members. When you do this in the resultant page you see something like - "Search result for - your search query". Now if your social networking site is not safeguarded from xss attacks, attacker might inject a malicious script in the search query that will get executed when you do a search which will result in the attacker stealing your data. Worst case your authorization cookie is stolen and your account is compromised.

As stated in Wiki - A reflected attack is typically delivered via email or a neutral web site. The bait is an innocent-looking URL, pointing to a trusted site but containing the XSS vector. If the trusted site is vulnerable to the vector, clicking the link can cause the victim's browser to execute the injected script.

So when you get that mail again saying hey please review these bull**** documents please do not click it. It's either a spam or worst - attempt to steal your sensitive data, since you are logged into multiple sites with active sessions. Always see who is it sent from? Do you know that person? Does the link look suspicious? No one will give you money just like that or you will not be selected for a lucky draw without applying for it :). If you are curious as to what it actually is then copy the link and open it in incognito mode which have no active sessions or logins. Now if it asks you to login it's time to turn around.

Now what do we mean by  innocent-looking URL? The URL is infact that for a trusted website. All attacker will do is inject an XSS vector in one of the request parameters. Say search parameter like in example above. He will also encode script tags and other characters so that the victim user will not be able to guess it's the affected URL. Anyway well soon see a real example of this with working code and the fix. Now lets see persistent xss.

Stored or persistent XSS attack

This is even worst that then the reflected one because this xss attack vector is stored on websites server (perhaps in it's persistent DB) and it will get executed every time user navigates to the section that uses this xss affected code. Lets see an example.

Consider a forum used to discuss programming related QnA. Again if the site is not safeguarded against XSS attacks then an attacker can ask a question which seems normal and legit but at the end contains a malicious script/attack vector. Now when he publishes the question it gets stored in the sites persistent storage and when any user tries to visit this question this malicious script will get executed on his/her browser and again your sensitive data can be stolen.

It's not always about the victims sensitive data though. An attacker can use XSS attack to permanently change the websites appearance or change display texts. Consider the damage it can do. For eg. attacker can change the announcement made by a reputed company which can result in it's stock prices crashing.

Enough with the theory let's see a practical example -

Testing for XSS

Code for this demo is available on my git repository  -
It's a web app. So you can download it and run it in a web server like apache tomcat. For this demo I will use my local setup.

NOTE : This web project has multiple workflows. The files you are interested in are -
This demo also assumes you are familiar with webapps, JSP, Java, Spring etc. I have written multiple posts on these topics in the past. You can search those in this blog if you are interested.You can find everything starting from configuring and using ivy dependency management to developing sample Spring application to deploying apps on tomcat. Above web app uses all these.

Once you have deployed the web app you can open it in your browser. It should look like following -

You can add normal text in the search box and do a "Search" or a "Search Xss Protect". Everything should work fine. For eg. when I input - "I am groot!", it just shows me the query I have input -

Now lets do "Search Again" and perform a normal "Search". But this time we will search differently. Enter following query in the search box -
  • I am groot!<script>alert(10)</script>
The result is -

As you can see we do see our normal result but we also see a prompt with 10 in it. Now if you revisit our input query we added a script tag at the end and the browser executed it. This was just for the demo, attackers script will execute silently in the background without prompts and steal/modify your data.

Now lets repeat the same search with text -
  • I am groot!<script>alert(10)</script>
Only this time you press "Search Xss Protect". What do you see?

What happened here? No prompts? That's because we escaped out input so that it is not harmful which is exactly what the fix is. You need to escape your user inputs so that they are not executed by your browser as scripts. Now that you have seen a demo of how XSS behaves and how you could safeguard it lets see how it actually works from code perspective.

Code working

You need to revisit the JSP file -
The main culprit line here is -
  • Your input : <%=query %
This essentially evaluates the query from Java. If this has script tags those will be evaluated too. What do we do? We escape it before this line is reached which is precisely what the code does when you press protected search -

        String xssProtect = request.getParameter("xssProtect");
        if(xssProtect != null && xssProtect.equalsIgnoreCase("yes")) {
            query = ESAPI.encoder().encodeForJavaScript(query);

NOTE : xssProtect is something I have internally used to differentiate between normal search and protected search for demo purposes. You need not worry about it. In production you should never have a normal search like scenario.

NOTE : ESAPI is a library that OWASP (Open Web Application Security Project) recommends. You can see a bunch of links in the related links section below. But you can technically use any library that escapes html like coverity security lib.

NOTE : Above was an example of reflected or non persistent XSS attack. What if we were storing this search queries in our internal DB and showing search history for a user? In that case it would be a persistent XSS attack.

Though above demo was a means to show you how XSS actually works and how it can be safeguarded you should never code it this way. JSP should never have such logic. All of this should be in backend logic possibly your servlets and controllers.

Related Links

Sunday, 26 February 2017

Working with primitives in Streams


In one of the previous posts on Understanding Java 8 Stream API we saw basics of working with primitives. To reiterate - there are 3 types of primitive streams -
  1. IntStream: Used for the primitive types int, short, byte, and char
  2. LongStream: Used for the primitive type long
  3. DoubleStream: Used for the primitive types double and float
These primitives are special they got their own Optional classes , their own functional interfaces. With our previous understanding in place we will see this new working with primitives.

Why primitive Streams?

Lets say we have a list of integers and we want to compute its average. How would you do it with streams.

    public static void main(String[] args) {
        List<Integer> myIntList = Arrays.asList(1,2,3,4,5);
        Optional<Integer> sum =,y) -> x + y);
        long count  =;
        double avg = sum.get()/count;

Output : 3.0

Well sure based on what we know so far we aced this. But do you see any drawback in above approach?
  • You are creating stream twice from the list - once to get sum and once to get count.
  • Then you are dividing it to get the average.
Though this works Java provides more optimized solution and this is where IntStream comes into picture.

    public static void main(String[] args) {
        List<Integer> myIntList = Arrays.asList(1,2,3,4,5);
        OptionalDouble intAvg =>x).average();

Output : 3.0

That's the one liner with IntStream. mapToInt() intermediate operation gives you a IntStream which has predefined operations like sum(), average() etc. Couple of points to note -
  • mapToInt() intermediate method returned IntStream
  •  average() terminal operation on IntStream returned a OptionalDouble. This is a new class. Notice it is not Optional<Double> but entirely a new class. 
  • Also note the method to get double value from  OptionalDouble class. It is getAsDouble() and not get() like you would have seen in Optional class.
  • Recollect out discussion before primitives have their own streams eg. IntStream , own Optional classes eg. OptionalInt and even functional interfaces eg. BooleanPredicate. But that's for later.
  • sum() on IntStream returns int instead of OptionalInt because if there are no elements in the Stream it would return 0.
    public static void main(String[] args) {
        List<Integer> myIntList = Arrays.asList(1,2,3,4,5);
        int sum =>x).sum();

Output : 15

NOTE : If you want Object stream back from primitive streams you can use mapToObj intermediate method.

Creating primitive streams

Now that we have basics out of our way let see how can we create a primitive stream. One method we already saw by using mapToInt() method on a normal Stream. Similarly you have mapToLong() and mapToDouble() methods to get LongStream and DoubleStream respectively. 

Other normal methods that we say in Stream class like of() or generate() or iterate() still apply to primitive streams as well.

    public static void main(String[] args) {
        IntStream.generate(() -> 1).limit(4).forEach(System.out::print);
        IntStream.iterate(1,(x) -> x+1).limit(4).forEach(System.out::print);

Output :

Some other methods are 
  • range(int startInclusive, int endExclusive)
  • rangeClosed(int startInclusive, int endInclusive)
    public static void main(String[] args) {
        IntStream.range(1, 5).forEach(System.out::print);
        IntStream.rangeClosed(1, 5).forEach(System.out::print);

Output :

Example demonstrates for InStream but same applies for LongStream and DoubleStream as well.

NOTE : Notice the range limits. For range end limit is exclusive where as for rangeClosed it is inclusive.

Understanding summaryStatistics

Now lets say we have an IntStream and we need to to output difference between the max element and min element in the stream. How would you do that?

    public static void main(String[] args) {
        List<Integer> myList = Arrays.asList(1,2,3,4,5);
        int max = -> x).max().getAsInt();
        int min = -> x).min().getAsInt();
        int difference = max - min;
        System.out.println("Difference : " + difference);

Output :
Difference : 4
which is expected since 5-1 = 4.
But again we need to create stream twice and need two terminal operations to get result. Java helps us here too with summaryStatistics method.

    public static void main(String[] args) {
        List<Integer> myList = Arrays.asList(1,2,3,4,5);
        IntSummaryStatistics intSummaryStatistics = -> x).summaryStatistics();
        int difference = intSummaryStatistics.getMax() - intSummaryStatistics.getMin();
        System.out.println("Difference : " + difference);

Output :
Difference : 4

Related Links

t> UA-39527780-1 back to top