Spring data : Execute the query without even writing it!

Hi there, I am writing after a long time, couldn’t write for last few months due to various circumstances. Anyway, I am back now and I am also celebrating ‘no guffs in the blogs day’ today. So, let’s come straight to the point. We will discuss about the newly released spring data framework. The framework which has made database interaction a piece of cake.

Spring data provides support for almost all kinds of databases which exist in this world. We will discuss about relational and non-relational (mongodb) databases in this blog. First, let us discuss a bit of theory of how spring data works. We will then see some examples. Spring data mainly relies on two things for database communication namely Bean class and repository. Bean class is the mapper class for the database table (or mongo collection) and repository is just an interface. Spring data provides many types of repositories, e.g. Crud Repository, Mongo repository to name a few. It uses the reflection and methods defined in user repositories to build and execute the queries. Well, this may seem a bit complicated. Let’s see the whole process step by step.

Suppose we have an Employee bean, defined as follows:

@Entity

public class Employee {

      @Id

      @GeneratedValue(strategy=GenerationType.AUTO)

      private int id;

      private String name;

      public int getId() {

            return id;

      }

      public void setId(int id) {

            this.id = id;

      }

      public String getName() {

            return name;

      }

      public void setName(String name) {

            this.name = name;

      }

}

Remember that, it is essential to have setter and getter methods in order for spring data to work properly. And, the setters and getters should abide java bean standards as well.

In above example, @Entity annotation tells us that this is an entity, @Id indicates id and @GeneratedValue is self explanatory (more on annotations here). Apart from this, we should tell spring where to search for entities and repository in configuration file. Examples of which are shown here. Now let us see the the repository.

public interface EmployeeRepository extends CrudRepository(Employee, Integer){

}

Please note that Repository is always an interface. We create our own interface which extends Spring’s interface (CrudRepository in our example). Spring will implement our interface and will provide support for different db operations. Methods supported by Crud repository are listed here. Now, let’s see how we can persist the record. Below is the code.

Employee employee = new Employee();

employee.setName("abcd");

EmployeeRepository repository = context.getBean(EmployeeRepository.class);

int id = repository.save(employee);

See the magic, we haven’t defined save method anywhere and still, we are able to save employee record. Save() will return the id of newly created record which we can use further. Context is the reference of spring application context.

Now comes the most exciting part. Suppose we want to find the employees whose name is ‘MyName’. How should we do it? Should we write a query? Should we create criteria? Should we create a prepared statement? Well, as I told you earlier, we do NOT need to create any of these. All we need to write is, well, a mere line. Yes, just a line. We will modify our repository so that it includes a mehod. The modified repository will look like this:

public interface EmployeeRepository extends CrudRepository(Employee, Integer){

      public List<Employee> findByName(String name);

}

Observe the name of the method very carefully. It says findByName (B and N are capital). It has to be declared EXACTLY like this. There is no relaxation here. Spring starts scanning the method name from the right, uses reflection and string manipulations to locate member variables of entity class and relations between conditions. It then generates the query. However, all we need to do is JUST to define the method. This is it. Spring will take care of the rest. So, the code to execute the query or to fetch the employees with given name I should say, will look like this:

EmployeeRepository repository = context.getBean(EmployeeRepository.class);

List<Employee> list = repository.findByName("MyName");

Voila! We are done. Where is the query? Where is a criterion? Where is prepared statement? Where is result set? They are nowhere to be seen. Spring data saves us from these burdens and takes care of them. So, we have just executed query without even writing it! Isn’t it magic? It indeed is.

However, we need to be extra careful when defining the methods in repository. As I mentioned earlier, order, case and arguments and return type DO matter A LOT (i.e. above method returns only one employee if a return type is Employee object). Let’s see some example methods for different queries:

1. Find Employees whose name is ‘Name’ and id is greater than 100:

public List<Employee> findByNameAndIdGreaterThan(String name, int id);

2. Find Employees whose age is greater than 25, in an ascending order of age:

Interface method:

public List<Employee> findByAgeGreaterThan(int age, Sort sort);

Implementation:

List<Employee> list = repository.findByAgeGreaterThan(25, new Sort(Sort.Direction.ASC, "age"));

See? There is a Sort object which defines the sort direction and field. Similarly, there is a class for pagination. Let’s see an example of the same.

3. Find Employees whose age is greater than 25, in ascending number of age, show first 10.

Interface method:

public List<Employee> findByAgeGreaterThan(int age, Pageable pageable);

Implementation:

PageRequest request =

            new PageRequest(1, 10, Sort.Direction.ASC, "age");

List<Employee> list = repository.findByAgeGreaterThan(25, pageRequest);

For pagination, Spring data defines Pageable interface which in turn is implemented by PageRequest class.

This was overview of spring data and how repositories work. For mongo database (non reational), spring has MongoRepository which helps in inserting and querying the data. For in-depth knowledge of spring data and how the methods should be named, I would recommend to go through this and this links.

That is your lot for this time. Hope you have liked it. Will be back with a new topic as soon as I can.

Happy Coding.

~ Cheers

Features of Dolphin and Spider

“What the hell is he up to?” You may think. “Why on earth is he going to discuss the animals over here? Is he out of his mind?  What the f..”  Wait. Before you go too far down that road and possibly plan my murder, let me clarify this. We are not going to discuss about a mammal and an arachnid here. Dolphin and Spider are the code names of Java 7 and Java 8 respectively. Although code names have been dropped since java 8, spider is the known name among the Javaites.

Java 7 was released on 28^th July 2011 and Java 8 was released on 18^th March 2014. Even though Java 7 was released 2 and half years ago, everyone hasn’t migrated to it. People are gradually migrating to it though. If I talk about my team then, we are still evaluating Java 7 and its features and compatibility of our application to it and are planning to migrate in a month or two. Hence, going by this way, I think it will take very long for people to switch onto Java 8, and it isn’t supported on windows XP (There are workarounds though) which I believe, will be a significant factor in delay. But the features it brings are commoving. Let’s discuss them. We will discuss some of the features of Java 7 and 8. One more thing, we will not discuss the language changes (e.g. Strings in switch in Java 7), because, well, every Tom, Dick and Harry is aware about those. We will discuss the features/APIs which are added in Java platform.

Java 7 features:

• File I/O library: Java 7 contains Java.NIO 2.0 package (NIO stands for non-blocking I/O). It contains different classes which help you play around with file systems very easily. The ways to access the file have become very much compact and optimized (i.e. you can read the file in just 2 lines of code rather than the old school method of creating a file object, then opening file input stream then reading the file line by line or so). This package even provides the facility using which you can create/access the custom file systems. One of the examples is Zip file provider, which can treat the zip file as a file system and allows you to access the different files from it. Isn’t this awesome? Well, for deeper dive into this, I would recommend to visit this link.
• Use of Timsort algorithm: : In previous versions, merge sort was used to sort the arrays. Merge sort works by splitting the main array into multiple arrays (each containing 1 element), and then merge them and sort them to finally produce the array which is in sorted form. Timsort was invented in 2002. It works in a different way. It first searches the array for the segment which is already sorted. It removes such sub array from main array (to make the sorting of remaining elements easier). It against performs same task on the remaining elements. Such iteration is called a ‘run’.
  Although both merge sort and Timsort have equal worst case complexity (O(n logn)), Timsort provides faster best case performance (O(n)) compared to that of merge sort (O(n logn)). Also, data analysts assume that, in real time systems, an array contains many such sub arrays which are already in sorted form, making the task of Timsort easier. Hence, in Java 7, it is utilized. More on this is described here.

• Support for dynamism: First, let us discuss what dynamic language is. A dynamic language is the one that can exhibit the behavior at runtime which the other languages can exhibit at compile time only. Let’s have an example to simplify this. Java is a static language, meaning, the operations like type checking of methods (e.g. a method with an integer argument can’t be called using string argument) happen at compile time. Once the compilation is performed and byte code is generated, it is done and dusted. You can’t modify this method at runtime. Whereas in dynamic languages (like Python, Ruby), it is possible. You can modify the behavior of method at runtime; you can even make a class extend another at runtime. Due to this, there has been an increasing pressure on Java creators to include dynamism in Java and hence, they have included some of it in Java 7. This may be a long shot for us to understand; however, to get more idea of it, I would recommend visiting this link.

Well that’s it for Java 7; let’s move onto Java 8 then.

Java 8 features:

• Support for Lambda expressions: This is the feature I was waiting so eagerly to discuss about. This provides those powers to Java that it lacked compared to the other high level languages. It provides the wings of functional programming to Java. Let us discuss about functional programming first.
  Functional programming is a programming in which the functions (and not objects) are basic building blocks of a program. It helps very much in reducing the complexity of the language and makes tackling complex problems easier. Javascript is a functional language. Example of the same is provided here.
  Now, as we all know, Java is an object oriented language (except for the primitives, obviously). Each and everything in Java has to be accessed through the objects and classes (in case of statics). The methods (functions) cannot live on their own; they must be called using either objects or classes. Hence, we can say that functions are not important in Java, or, to put it in the other words, functions are less important than objects in java.
  In any functional language, functions are THE most important parts. I would even say that functions are the VIPs of any functional language. The most important feature of any functional language is ‘Closure’, which, according to wiki, is a function or reference to a function together with a referencing environment—a table storing reference to each of the non-local variables of that function. Sounds too complex, doesn't it? Anyway, let’s not go much into this definition. Let’s talk about how java achieves this purpose.
  
  The feature in Java 8, which is similar (or let’s say, closest) to closure is, Lambda Expressions (LE). An LE basically means a function without declaration (similar to anonymous inner class). Following are the examples of LE:
  
  (int a, int b) -> {return a+b;}
  
  
  () -> System.out.println("Hello World");
  
  Doesn’t this seem interesting? Furthermore, LE can also be assigned to the reference of a functional Interface. A functional interface is an interface which contains just 1 method (e.g. Runnable). Following are examples of LE assigned to reference of functional interfaces:
  Runnable r = () -> System.out.println("hello world");
  
  
  new Thread(
         () -> System.out.println("hello world")
  
  

  ).start();

  
  This way, we can start a new thread without even overriding the run() method (we have LE’d the run() method here).
  
  Using an LE, we can even add default and static methods to the interfaces with concrete implementations. One can write a book on LEs, however, we will wind our discussion up regarding LE. If you want to study more, you can visit this link.
• Removal of Permanent Generations: Well well well, this also sounds like a commoving feature. Doesn't it? Here, we assume that everybody is aware about the permanent generation. If you don’t then, I would strongly recommend you to read this blog of mine first and come back. Go ahead, push this line in your stack, go and read the blog, pop this line back and continue reading.
  Back to our point, this feature sounds like living without the limits. This means you will no longer be haunted by the demon called ‘java.lang.OutOfMemoryError: PermGen space’. What a respite it is! The arguments named PermSize and MaxPermSize will be ignored. A warning will be issued if they are present, saying that Java doesn’t give a damn about them.
  JVM of Java 8 will be using native memory to store the classes’ data. That memory is called Metaspace. Native memory simply means memory available to Java process and is controlled by OS. However, using a parameter named MaxMetaspaceSize, we can limit the memory usage. If we don’t specify, Metaspace will grow dynamically at runtime depending upon usage. More on this is explained here.
• Date and Time API: Although existing versions of Java provide a bunch of APIs to play around with Date and Time, handling of these has always been tricky. Sometimes, we may need to write our own code to achieve the expected result even though there are plenty of methods available (one of the prime examples of it, that I faced recently, was to compare two calendar objects without considering time).
  As a part of Java 8, a whole new bunch of Date and Time APIs will be introduced into the mix (the means, now we will have two bunches). It will provide a lot more flexibility to manipulate Date and Time. We will not discuss these in more details as there are just classes and methods to get used to (in short, nothing commoving). However, if you want to study, you can study here. Also, as per readme file of these specifications, these APIs are not incomplete and may get changed later.
  

That’s that then. We have gone through some of the features of Java 7 and 8. I do accept that the features are described in (very) brief and we haven’t described all of the features (this again goes back to that ‘commoving’ filtration criterion). Each feature will need a separate blog to be discussed. Whenever I have enough information/knowledge about these features, we will discuss these thoroughly. Till then.

See you soon.

Java Collection Comparator : Compare two collections without iterating

Alright then, I am back, back to the world of blogging. It has been more than 3 months since I wrote something. You may ask why there has been so much delay. Well, I do not have any excuse for that (unless this can be considered as an excuse. Well, I do not have any excuse itself is an excuse, isn’t it? It is a virtual excuse, just like virtual memory of CPU. It is there as well as not there. Heh! Anyway, it seems I dragged this bracket on for too long, time to complete it.). You can say that I haven’t come across anything extra ordinary enough to be shared on blog. There you go. I have found an excuse for this, a physical excuse (rather than a virtual one which we discussed earlier). You can term this as Random Access Excuse (RAE) or Read Only Excuse (ROE) or excuse of any other type (as long as it is not virtual excuse). It depends upon you.

Enough of bakchodi I guess. Let’s switch onto the main topic of the blog. In this blog, we will discuss about comparison of two Java collections. Some of you might say that comparing two collections in Java is a piece of cake. What did you see extra ordinary enough in it to share? Well, I would say that we will compare the collections the easiest way. We will not iterate any collection, we will not even look into them, heck, we won’t even know the elements of collections (i.e. we won’t know whether they will contain Integers, Strings or any user defined objects)! It will be totally generic comparison. Does it sound interesting now? I guess yes. Let’s move ahead then.

Have you ever come across the use case in which, say, you are displaying a list of 20 items on UI, then the user edits some of the items (say 5) and clicks on save button. Then, in the business layer (or service layer), you will have both the lists (changed and unchanged one. (You can either store the unchanged list on session or grab it back from database.)). From these 2 lists, you will have to figure out which items got changed so that you can issue database update queries on them.  For this, you will have to iterate the lists, compare the Items (possibly ItemDTO objects) and generate the third list which contains only the changed ones. You know how painful it can be to write the code for this? Don’t worry. Our solution is tailor made for such problems.

The solution needs 3 values from the consumer. They are described below:

• Source collection (i.e. the one coming from UI)
• Second collection (i.e. the one which the source collection needs to be compared against)
• Reference of result collection (the difference will be populated into this collection)

We are performing the following steps in our comparison. We will use the terms first collection, second collection and third collection to refer the respective inputs:

1. Create two empty Sets i.e. first set and second set.
2. Add all the elements of first collection into first set.
3. Add all the elements of first collection into second set.
4. Call retainAll method on second set. Pass second collection as an argument. E.g. secondSet.retainAll(secondCollection).This method retains the common elements into the collection which invoked it (i.e. intersection). More on this is described here. Hence, after the execution of this step, first set will contain all the elements of first collection and second set will contain the common (unchanged in our case) elements from both the collections.
5. Call removeAll method on first set. Pass second set as an argument. E.g. firstSet.removeAll(secondSet). This step will remove all the common elements from first set. In our case, all the Items which are unchanged from UI will be removed. More on removeAll is described here.
6. Populate the third collection with the elements of first set.
7. Voila! We are done. And this is not actually a step.

Some might ask why we need the third collection here. Why the method just doesn’t return the result? The answer being, well, the method accepts the Collection references. Hence, it won’t know what is passed to it at runtime. You can pass the List, Set or even a Queue (i.e. everything that extends Collection interface). Hence, the decision of which collection you want the difference to be populated into is left to you (e.g. if you want to compare two lists and want a difference in a List/Set/Queue, you can pass the empty List/Set/Queue in the third argument).

It is worth mentioning that when you are passing the list of user defined objects to this method, you MUST override equals and hashcode methods in your class. The solution does not have any comparison logic. It relies on comparison logic of Set, and Set internally utilizes equals and hashcode methods to differentiate the objects. If you do not override these, all the objects will be marked as different even though all the attributes have same values (the reason being, if a class does not have its own equals and hashcode methods, same of Object class are utilized and they generate unique hashcode for each and every object. Hence, if hashcode is different, the objects are classified as different). More on equals and hashcode is explained here.

That’s that then. I have uploaded the source code with couple of comparisons. You can download it from here. Hope it helps you somewhere down the line. Also, a big thanks to Harshal Choksi for bringing up this scenario and providing the inputs whenever needed. And yes, I will try to be back quicker for the next blog.

~ Cheerio.

The dreaded Production Outage

Outage. The word which sends shivers down our spine when me or my team mates hear about it. It is one of the most fearful words of the software industry. It can vanish all of your joy/confidence in a heartbeat. Just when you think you are doing good and gaining the confidence/trust of your client, suddenly, out of nowhere, in a pleasant morning, you will see an email from your client (with a little red exclamation mark) saying that they are facing an outage in the production environment. Generally, it will be followed by a couple of escalation emails from your manager as well as leader. Bang! All your good deeds are as good as nothing now. The misery won’t stop here; you will be immediately called into an emergency meeting. In this, you will get your ass whipped and will be asked to find the root cause of this outage and provide a lengthy, Shakespeare style email (with all the decorations and formatting) to the client. And, all you will have for analysis (i.e. data and information) will be, well, nothing! This is the subject of this blog. Let’s start with it.

There are multiple definitions for outage. Assume that yours is a web application which is hosted on so and so server. One instance may refer to the server crash (due to any hardware/software failure) which would make all requests get rejected with http 4XX (or any similar) error message. One instance may refer to the crashing of the application hosted on server, resulting in http 5XX error message. One instance may refer to database crash, which will make all the operations fail with ‘technical error’ (or similar) error. One instance (which we will emphasize more in this blog) may refer to the application becoming unresponsive. This is very mysterious instance. The server seems to be working fine, the application seems to be working fine, the database seems to be working fine but still, when someone sends request to server, the browser will display ‘processing’ symbol and then it will time out eventually. See? I told you it is much deeper than it seems.

We faced similar situations multiple times in last month. The client kept on saying that the services became unresponsive and they had to restart the server to make the services available again. For our analysis, we asked for server logs, error logs, database logs, application logs, call logs, message logs, Whatsapp logs and any other type of logs which may exist on this planet. We got all the logs in some time and found nothing. Absolutely nothing! We were certain that there was not any hardware/server/database fault. After some googling and analysis, we found that there must be something wrong with JVM. Hence, we asked them to provide the heap dump of the JVM of production environment.

Let me explain about the heap dump first, well, not exactly heap dump. Let me start with the heap first. Heap (or I should say, heap memory) is an area which is utilized by JVM to allocate the memory for newly created objects. Say I have written a line in java program like Date d = new Date();. This will create a new Date object. That object will be stored in the heap memory. It will be alive as long as we have at least one reference (in our case, d) in our program. When all the references are destroyed, the object will be garbage collected by garbage collector (more on that later). Heap has two parts mainly:

• Eden Space: This is the part of the heap where the newly created objects will be stored. The memory for all the new objects is initially allocated from this part. In our life, Eden space refers to school.
• Survivor Space: When JVM ‘thinks’ that young gen space is almost full, it invokes garbage collector on it. GC cleans up all the objects which do not have any valid reference in the executing program(s). The objects which survive this garbage collection are moved to Survivor space (a reward for their so called achievement!). In our life, Survivor space refers to college. Ironically, our days to survive start from college too!

              Both Eden and Survivor space are parts of Young Gen space.

• Tenured Generation: This area contains the objects which are alive for quite a long time. Examples of such objects are static objects, instance variables of servlet class etc. It is also known as Old Gen space. In our life, this refers to the place where you are probably sitting and reading this blog i.e office.

That was all about heap. Before we start with heap dump, let me explain one more term:

• Permanent Generation:  When we create an instance of a class or let’s say, when we call any static method/member of a class, the respective class gets loaded in the memory.  In our previous example, Date class will be loaded in the memory before Date object is created. Permanent Generation space is the space where all the loaded classes are stored. Permanent Generation is not a part of heap memory. I couldn't find any mapping for this space in our life. Suggestions are welcome.

Now, let’s start with the heap dump. A heap dump is a binary or textual representation of Java heap which is written to a file. It is like a black box of the air craft. A heap dump has all the information about the objects present in the heap, their object tree and the memory occupied by them. There are many tools available in the market which can read such heap dumps and create the object charts with all the statistics. MAT plugin for eclipse is the popular and free tool. Another licensed tool is YourKit. JVisualVM (comes packaged along with java 6) is also very useful tool in heap dump analysis.

Back to our story, when we got the heap dumps, we started analyzing them. During our analysis, we saw a couple of large stringbuffer objects into it (around 3 GB each). They were occupying 6 GB. Also, there were many other heavy objects which ate up to 2 GB. In the production environment, JVM heap size was configured to 8 GB. All these large objects ate up all the memory, leaving no room for creation of any new object. Hence, the whole system went into comma and stopped responding the requests. The mystery was finally solved!

This was not the end though; identifying the problem was just a small step. We had to find the root cause of generation of that object and eventually, provide a patch which would prevent such situation from occurring. We analyzed that and it was one specific case of looping which appended characters to string buffer object (more on this later). But the presence of large objects in heap dump spawned a very interesting question: Java has much applauded Garbage collection algorithm. What the heck is it and why it was not able to clear these large objects?

Main task of Garbage collector is to clean up the objects which are not needed by the program. The criteria to decide which objects are ‘not needed’ is very complex. There are plenty of algorithms for garbage collection. Detailed explanation for the same is provided here. Instead of going into more details of those algos, we will focus on how garbage collector works with the JVM. There are 2 types of garbage collections:

1. Concurrent GC: Concurrent garbage collector uses a single thread which runs concurrently with the java application. The main motive of concurrent generator is to prevent tenured generation space from filling up. When concurrent GC is unable to reclaim the memory, full GC is triggered.
2. Parallel GC or Full GC: Parallel garbage collector belongs to the family of stop the world collector. It won’t start until old generation space runs out of memory. But as soon as that happens, Full GC starts, it will cause all the application thread to pause, resulting in unresponsiveness of the application.

In the production environment, Full GC is configured to kick in when old gen space is 70% used. When that happens, Full GC will kick in and CPU spike will touch 100% mark, causing all the requests to halt.

Back to our string buffer problem, we were storing a string buffer object in user’s session. We were performing some operations on it which used to result in increase/decrease in the size of that object. However, in one specific scenario, such objects (in user’s sessions) used to grow indefinitely. As they were in session, they were live. And GC was not able to recollect them as garbage (obviously it shouldn’t!), this increase led to triggering of full GC and eventually, bringing down the whole application. What a misery!

  As it is very difficult to trace and fix such problems, I have provided some guidelines which one should consider while coding to avoid such disasters.

• Never store any heavy objects in session, store them in requests instead.
• While writing any recursive or looping method/logic, be extra careful. Test all the scenarios. In most of the cases, the method will work in 99 out of 100 scenarios. But, the remaining 1 scenario will come back to bite you in the back side when your application gets deployed on production.
• Be very careful while using third party jars. Use object caching/object pooling concepts in a thriftier manner. Gather enough information about the class loading/memory consumption of a third party library before implementing in the application.
• Execute the performance testing of each and every smallest possible feature of the application. There are plenty of free tools available in the market for performance testing (like BadBoy, JMeter, Load runner etc).
• It is always better to have 5 null pointer exceptions than 1 out of memory error in your code.
• There are plenty of such points. You can search on google. It seems tedious to me to write such points here.

Well, back to the original story (for the 135762432^nd time), we managed to trace and eventually fix that problem. The system then started working like a charm! The joy was back. The confidence was back. The trust was back. Things started to work very smoothly. However, the god of outage, sitting in a certain corner of the world didn't like that. Hence, he sent another one from his troops to say hi to us a month later.

And, here we are, battling against another outage. This time, even heap dump has refused to provide the enough information. Preliminary analysis says that the permanent generation space (does it ring any bells?) gets filled up with the classes which get loaded multiple times. This seems to be the whole new world. However, we are ready for the battle, with our weapons, to fire in the dark J

~ Au revoir!

Tortoise SVN Pre-commit hooks: Hook yourself to safety

Writing something after a long time, well; not that much long but relatively long I should say. This has been a fantastic week for me, both footballistically and technically. Footballistically, Arsenal won back to back matches this week. In the match played in midweek, they showed some scintillating football. It contained crisp passing, swift movements, darting runs, insane finish and gorgeous understanding between the players. It was like watching porn on a football pitch. Reaction of Arsène Wenger (manager) after the first goal was priceless! Let me tell you, he deserves each and every bit of it. The efforts he put in to create such team were immense. He created the whole empire right from the scratch and now he is enjoying the fruits. What a guy he is! The talent he is having is unparalled. (*deviating from topic alarm*) Let me come out of this fantasy and start discussing what I am here to discuss about. Change of topic is good sometimes by the way.

I said this has been a good week for me technically also. I developed a solution for a very specific and rare problem but it was very useful. Let me provide a preface first.

We are using Tortoise SVN as a version control tool. The repository resides at client side. Hence, whenever you commit something to the branch, it directly goes overseas and clients can see the changes. We are using JUnit framework for unit testing and hence, test classes also go overseas. In test classes, generally we have many crap things termed as ‘test data’. Those things include database urls, passwords, file paths, encryption keys etc. Before sending these classes overseas, we have to clear all these data.

In English, there is an idiom which says, ‘To err is human’. All the members working in my team are humans (FYI!). Well, not sure about one or two (including myself) but rest all are humans. Hence, we often forget to delete those test data before committing the files. In last week, for the umpteenth time, we received an email from client saying that our classes contained sensitive data and all. It was a typical client escalation. They said that in spite of saying the same 12672 times, they were still seeing sensitive data being committed. This spurred flurry of meetings and discussions on our side and we decided to introduce something which would prevent a developer from committing the file if it contains any sensitive information. 

I googled a lot about this and found an approach to prevent this. Tortoise SVN provides the facility to run our own scripts on various events like Pre Commit, Post Commit, Pre Update, Post Update etc. The most suitable event for me was pre commit event. What I did was; I created a java program which would search for particular text in each and every file which is being committed. It would throw an exception if a match is found which would make the commit process stop. These scripts are called hook scripts. There are 2 types of hooks: Server side hooks and client side hooks. Server side hooks are installed on server and they would work for all the commits whereas client side hooks need to be installed on each and every client machine. We did not have SVN server hence server side hooks were not of any use for us. We then decided to go with client side hooks.

Let me explain you how SVN works. When we check the checkboxes against file names and click on commit, SVN creates a .tmp file in Temp folder of user (considering windows machine). This file contains multiple lines. Each and every line of this .tmp file is full path of file which is being committed (i.e. if 4 files are being committed then .tmp will have 4 lines). SVN also supplies the path of .tmp file to hook script (there are other arguments also which are passed but we don’t require those). So, the script I wrote used to read this file line by line then, read the file present at that path. If it encountered any keyword (like password etc) then it threw exception resulting in failure in commit process. I created a batch script (.bat file) which called jar file (containing my code). Following are the steps to attach a hook script:

• Open Settings window of tortoise SVN and go to hook scripts as shown in the image below:

• Click on ‘Hook Scripts’. It will open hook scripts menu as shown below:

• In Hook Type, select pre commit.
• In the Working copy path, select the path where you have created the branch.
• In the Command Line to Execute, select the script which you want to execute (.bat file or .exe file).
• Check those two checkboxes. Click on ‘OK’ and ‘Apply’ and Voila! Your hook script is ready.

The only situation in which I think this script will create problem will be the one in which you are planning to commit large number of files, each of large size. It will take some time to check all the files. But, I guess the delay is any day better than receiving vitriol from the client resulting in kick up the back side.

I have attached sample batch file and jar file which searches for string “password” for all the files having extensions in (.java, .jsp, .js, .xml, .config, .properties and .txt). You just need to change the path in the .bat file which is executing the jar file. You can download it from here.

A special thanks to my colleague Niket Patel for helping me out on this. Cheers mate.

That's yer lot for this time. See ya.