Change the authentication error message based on Accept Header in WSO2 API Manger

When you send the request with Accept header and the api call failed due to authentication failure, by default WSO2 API Manager will return the application/xml as a response. However, If you want your Accept header to be honored in the authentication failure cases. In my sample, I have just introduced to handle JSON and default as XML.

Microservices Framework for Java By WSO2

Microservices Framework for Java implemented by WSO2 currently available for developer preview and very soon WSO2 will release the WSO2 Microservices Framework 1.0.0. Based on facts WSO2 Microservices Framework is lightweight high performance runtime implemented on Java for java developers who are looking for compressive framework.


WSO2 Microservice Framework code base is free and open source that available in WSO2 Github repository 

Fail safe Mode in WSO2 ESB / WSO2 API manger

By default WSO2 ESB/APIM servers will start in fail Safe mode by ignoring the failure of artifacts. Server will rename the failed artifact with .back file and move forward with rest of the artifacts. However there are situation where you do not want to deploy the server even a single artifact fails, This is typical situation in micro services concept. 


By disabling the fail safe mode in WSO2 ESB/APIM, it is possible not to deploy all the services even a single artifacts get fails during the deployment.


Add failsafe.mode.enable=false in WSO2_HOME/repository/conf/synapse.properties

Exposing the WSO2 Admin service as REST service

There can be a situation where you want to invoke the wso2 admin services from your external custom management tool. This post is taking a single use case where, making the WSO2 admin service for blocking users as a REST service.

WSO2 ESB send same request to different Rest services

In this scenario I need to send a post request to two different REST services, I am using REST API configuration of WSO2 ESB First I need to post a request to first service and based on successful posting then need to post this same original request to another service and also But I need to obtain the response from first service and send it to client.However I do not need to obtain the response from second service.

I have illustrate this scenario in the simple flow diagram.

Relevant Synapse configuration.

Remove the payload and send POST request to backend via wso2 esb

There are situations where back-end does not need a payload for POST request. However in wso2 ESB has the payload that mediate before the back-end call. In order to drop the message payload you need to perform two tasks in the wso2 ESB.

• Create empty payload using Payload mediator
• use the FORCE_POST_PUT_NOBODY property to enforce.

Example is given below that demonstrate the use case. 

Bind WSO2 Mamanement Console to a certain IP Adress

There are lot of Security measures when deploy the WSO2 Server products. One of them is to restrict the accessibility of the server for certain IP addresses.

In WSO2 serves, it is possible to restrict via the the tomcat valve.

Edit
/repository/conf/tomcat/context.xml 


Add
<Valve className="org.apache.catalina.valves.RemoteAddrValve" allow="127\.0\.0\.1"/ >

Connecting and monotoring JMX of WSO2 products in EC2 instance

Most of the time current deployments are hosted in EC2 instance as it is very reliable and scalable. When it come to monitoring the WSO2 products via jmx in EC2 instances. You have to add some parameters inorder to connect with it.


add below parameters in /bin/wso2server.sh

    -Dcom.sun.management.jmxremote \

    -Dcom.sun.management.jmxremote.port=XXXX \

    -Dcom.sun.management.jmxremote.ssl=false \

    -Dcom.sun.management.jmxremote.authenticate=false \

    -Djava.rmi.server.hostname=XXX.XXX.XXX.XXX \

Here port should be port that not used any where in the given instance and host-name should be private IP of the EC2 instances.

Access and monitor the server via Jconsole by JMX url connection that has the domain name and the jmx ports define in the carbon.xml  

Known errors and issue while Running ciphertool in WSO2

I have seen several user mistake and issues that cause the error while running ciphertool.sh of WSO2 carbon servers. So based on my previous experience, I have listed down the error that I encounter so far while using the tool and solution for that...


Error set 1

[vanji@vanjiTestMachine bin]# ./ciphertool.sh -Dconfigure

[Please Enter Primary KeyStore Password of Carbon Server : ]

Exception in thread "main" org.wso2.ciphertool.CipherToolException: Error initializing Cipher

        at org.wso2.ciphertool.CipherTool.handleException(CipherTool.java:861)

        at org.wso2.ciphertool.CipherTool.initCipher(CipherTool.java:202)

        at org.wso2.ciphertool.CipherTool.main(CipherTool.java:80)

Caused by: java.security.InvalidKeyException: No installed provider supports this key: (null)

        at javax.crypto.Cipher.chooseProvider(Cipher.java:878)

        at javax.crypto.Cipher.init(Cipher.java:1653)

        at javax.crypto.Cipher.init(Cipher.java:1549)

        at org.wso2.ciphertool.CipherTool.initCipher(CipherTool.java:200)

This error can cause when keyAlias miss match when generating the key-store, Therefore please reconsider to generate right Key-store with the right keyAlias OR change the values in carbon.xml

Error set 2

I have notice flowing IOError read error while working on windows machine

[Please Enter Primary KeyStore Password of Carbon Server : ]

Exception in thread "main" org.wso2.ciphertool.

CipherToolException: IOError read

ing primary key Store details from carbon.xml file

        at org.wso2.ciphertool.CipherTool.handleException(CipherTool.java:861)

        at org.wso2.ciphertool.CipherTool.getPrimaryKeyStoreData(CipherTool.java

:305)

        at org.wso2.ciphertool.CipherTool.initCipher(CipherTool.java:180)

        at org.wso2.ciphertool.CipherTool.main(CipherTool.java:80)

Caused by: java.io.FileNotFoundException: C:\Program Files\Java\jdk1.6.0_16\bin\

repository\conf\carbon.xml (The system cannot find the path specified)

        at java.io.FileInputStream.open(Native Method)

        at java.io.FileInputStream.(FileInputStream.java:106)

        at java.io.FileInputStream.(FileInputStream.java:66)

        at sun.net.www.protocol.file.FileURLConnection.connect(FileURLConnection

.java:70)

        at sun.net.www.protocol.file.FileURLConnection.getInputStream(FileURLCon

nection.java:161)

        at com.sun.org.apache.xerces.internal.impl.XMLEntityManager.setupCurrent

Entity(XMLEntityManager.java:653)

        at com.sun.org.apache.xerces.internal.impl.XMLVersionDetector.determineD

ocVersion(XMLVersionDetector.java:186)

        at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(X

ML11Configuration.java:771)

        at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(X

ML11Configuration.java:737)

        at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.

java:107)

        at com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(DOMParser.

java:225)

        at com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(Doc

umentBuilderImpl.java:283)

        at javax.xml.parsers.DocumentBuilder.parse(DocumentBuilder.java:180)

        at org.wso2.ciphertool.CipherTool.getPrimaryKeyStoreData(CipherTool.java

:289)

        ... 2 more

There is a windows long classpath issue in the script. Please edit the following lines in ciphertool.bat script 

[vanji@vanjiTestMachine bin]$ ./ciphertool.sh -Dconfigure
[Please Enter Primary KeyStore Password of Carbon Server : ]
Exception in thread "main" org.wso2.ciphertool.CipherToolException: Error initializing Cipher
        at org.wso2.ciphertool.CipherTool.handleException(CipherTool.java:861)
        at org.wso2.ciphertool.CipherTool.initCipher(CipherTool.java:202)
        at org.wso2.ciphertool.CipherTool.main(CipherTool.java:80)
Caused by: java.security.InvalidKeyException: Wrong key usage
        at javax.crypto.Cipher.init(Unknown Source)
        at javax.crypto.Cipher.init(Unknown Source)
        at org.wso2.ciphertool.CipherTool.initCipher(CipherTool.java:200)
        ... 1 more

Edit the line from 73 to 77 with the following lines

call ant -buildfile "%CARBON_HOME%\bin\build.xml" -q 
set CARBON_CLASSPATH=.\conf 
FOR %%c in ("%CARBON_HOME%\lib\*.jar") DO set CARBON_CLASSPATH=!CARBON_CLASSPATH!;".\lib\%%~nc%%~xc" 
FOR %%C in ("%CARBON_HOME%\repository\lib\*.jar") DO set CARBON_CLASSPATH=!CARBON_CLASSPATH!;".\repository\lib\%%~nC%%~xC" 



Error Set 3

[vanji@vanjiTestMachine bin]$ ./ciphertool.sh -Dconfigure 
[Please Enter Primary KeyStore Password of Carbon Server : ] 
Exception in thread "main" org.wso2.ciphertool.CipherToolException: Error initializing Cipher 
        at org.wso2.ciphertool.CipherTool.handleException(CipherTool.java:861) 
        at org.wso2.ciphertool.CipherTool.initCipher(CipherTool.java:202) 
        at org.wso2.ciphertool.CipherTool.main(CipherTool.java:80) 
Caused by: java.security.InvalidKeyException: Wrong key usage 
        at javax.crypto.Cipher.init(Unknown Source) 
        at javax.crypto.Cipher.init(Unknown Source) 
        at org.wso2.ciphertool.CipherTool.initCipher(CipherTool.java:200) 
        ... 1 more 

If you are changed the default keystore privided with wso2server new one, make sure you have change all the references for that keystore. You may have to change the entries in following files. 

WSO2Server/reposotory/conf/carbon.xml 
WSO2Server/repository/conf/security/secret-conf.properties 
WSO2Server/repository/conf/sec.policy 
WSO2Server/repository/conf/security/cipher-text.properties 
WSO2Server/repository/conf/tomcat/catalina-server.xml 
WSO2Server/reposotory/conf/axis2/axis2.xml 

Not only the keysore name, make sure you change keypassword, keystore pasword and keyalias according to your keystore.

Error Set 4


[vanji@vanjiTestMachine:~/software/wso2/wso2esb-4.8.0
$ sh bin/ciphertool.sh -Dconfigure
Exception in thread "main" org.wso2.ciphertool.CipherToolException: IOError reading primary key Store details from carbon.xml file 
at org.wso2.ciphertool.CipherTool.handleException(CipherTool.java:861)
at org.wso2.ciphertool.CipherTool.getPrimaryKeyStoreData(CipherTool.java:305)
at org.wso2.ciphertool.CipherTool.initCipher(CipherTool.java:180)
at org.wso2.ciphertool.CipherTool.main(CipherTool.java:80)
Caused by: java.io.FileNotFoundException: /home/vanji/software/wso2/repository/conf/carbon.xml (No such file or directory)
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.(FileInputStream.java:120)
at java.io.FileInputStream.(FileInputStream.java:79)
at sun.net.www.protocol.file.FileURLConnection.connect(FileURLConnection.java:70)
at sun.net.www.protocol.file.FileURLConnection.getInputStream(FileURLConnection.java:161)
at com.sun.org.apache.xerces.internal.impl.XMLEntityManager.setupCurrentEntity(XMLEntityManager.java:651)
at com.sun.org.apache.xerces.internal.impl.XMLVersionDetector.determineDocVersion(XMLVersionDetector.java:186)
at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:772)
at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:737)
at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:119)
at com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(DOMParser.java:232)
at com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(DocumentBuilderImpl.java:284)
at javax.xml.parsers.DocumentBuilder.parse(DocumentBuilder.java:180)
at org.wso2.ciphertool.CipherTool.getPrimaryKeyStoreData(CipherTool.java:289)
... 2 more



When you run the ciphertool.sh from outside the bin folder  you will see this error and this is the limitation of the tool.


I have listed the issue that i have encountered so far, if i found anything new I will keep update this blog-post with my new findings

Write the logs into External database in WSO2 Servers

Some time data mining purpose storing the logs in the database important and it is possible to do that with wso2 carbon products as well.

To achieve above task follow the steps that mention below. I have used mysql for demonstrate this task and it is possible to use and other RDBMS for this.

1. If the server is already running, stop the server.

2. Configure the database (say, LOG_DB) and create the following table (LOGS)

CREATE TABLE LOGS( USER_ID VARCHAR(20) NOT NULL, DATED   DATETIME NOT NULL, LOGGER  VARCHAR(50) NOT NULL, LEVEL   VARCHAR(10) NOT NULL,MESSAGE VARCHAR(1000) NOT NULL);

3. Configure the log4j.properties in the /repository/conf/

Since log4j.rootLogger is already defined append “sql” in it as follows.

log4j.rootLogger=ERROR, CARBON_CONSOLE, CARBON_LOGFILE, CARBON_MEMORY, CARBON_SYS_LOG, ERROR_LOGFILE, sql

Add the following,

log4j.appender.sql=org.apache.log4j.jdbc.JDBCAppender

log4j.appender.sql.URL=jdbc:mysql://localhost/LOG_DB

# Set Database Driver

log4j.appender.sql.driver=com.mysql.jdbc.Driver

# Set database user name and password

log4j.appender.sql.user=root

log4j.appender.sql.password=root

# Set the SQL statement to be executed.

log4j.appender.sql.sql=INSERT INTO LOGS VALUES ('%x', now() ,'%C','%p','%m')

# Define the xml layout for file appender

log4j.appender.sql.layout=org.apache.log4j.PatternLayout

4. Download the mysql driver from, http://dev.mysql.com/downloads/connector/j/5.0.html and place the jar (mysql-connector-java-5.1.31-bin) inside /repository/components/lib/

5. Start the server, you will be getting the logs in the LOGS table as well.

GIT 101 @ WSO2

Git

Git is yet another source code management like SVN, Harvard, Mercurial and So on!

Why GIT?

Why GIT instant of SVN in wso2?

I do not know why! it might be a off site meeting decision taken in the trinco after landing with adventurous flight trip ;)

• awesome support for automation story
• Easy to manage
• No need to worry about backup and other infrastructure issues.
• User friendly
• Publicly your code reputation is available.

GIT in WSO2.

WSO2 has two different repository.

• Main Repository.
• Main purpose of this repository maintain the unbreakable code repository and actively build for the continuous delivery story incomprated with integrated automation.
• https://github.com/wso2
• Development Repository.
• Development repository is the place teams play around with their active development.
• https://github.com/wso2-dev
• wso2-dev is a fork of wso2 repo!

UPDATE

Now this statement invalid as WSO2 has changed it process on Dec/2014

Rules

1. Developer should not fork wso2 repo.
1. Technically he/she can but the pull request will not accepted.
2. If something happen and build breaks! He/She should take the entire responsible and fix the issue and answer the mail thread following the build break :D
2. Developer should fork respective wso2-dev repo.
1. He/She can work on the development on her/his forked repo and when he/she feel build won't break he/she need to send the pull request to wso2-dev.
2. If pull request should be reviewed by respective repo owners and merge.
3. On the merge, Integration TG builder machine will get triggered and if build pass no problem. If fails, He/She will get a nice e-mail from Jenkins ;) so do not spam or filter it :D. Quickly respective person should take the action to solve it.
3. When wso2-dev repository in a stable condition, Team lead/Release manager/ Responsible person  has to send a pull request from wso2-dev to wso2.
1. WSO2 has pre-builder machine to verify the pull request is valid or not.
1. if build is passed and the person who send a pull request is white listed the pull request will get merged in the main repository.
2. if build fails, the pull request will be terminated and mail will send to the respective person who send the pull request. So now, respective team has to work out and fix the issue.
3. Build pass but not in whitelist prebuild mark it a need to reviewed by admin. But ideally admin will close that ticket and ask the person to send the pull request to wso2-dev ;)
4. If everyting merged peacefully in main repo. Main builder machine aka continuous delivery machine  build it. If it is fail, TEAM need to get into action and fix it.
4. You do not need to build anything in upstream, ideally everything you need should fetched from the Nexus.
5. Allways sync with the forked repository

GIT Basics

1. Fork the respective code base to your git account
2. git clone github.com/wso2-dev/abc.git
3. git commit -m “blha blah blah”
4. git commit -m “Find my code if you can” -a
5. git add myAwsomeCode.java
6. git push

Git Beyond the Basics

• Sych with upstream allways before push the code to your own repository

WSO2 GIT with ESB team

ESB team owns

1. https://github.com/wso2-dev/wso2-synapse
2. https://github.com/wso2-dev/carbon-mediation
3. https://github.com/wso2-dev/product-esb/

Nobody else other than in ESB team has the mergeship :P for these code repository. So whenever somebody try to screw our repo, please take a careful look before merge!

The first principle is no one suppose to build anything other than currently working project.

Good to read

http://www.vogella.com/tutorials/Git/article.html

[Architecture] Validate & Merge solution for platform projects

Maven Rules in WSO2

Please find POM restructuring guidelines in addition to things we discussed during today's meeting.  

1. Top level POM file is the 'parent POM' for your project and there is no real requirement to have separate Maven module to host parent POM file.
2. Eliminate POM files available on 'component' , 'service-stub' and 'features' directories as there is no gain from them instead directly call real Maven modules from parent pom file ( REF - [1] )
3. You must have a    section on parent POM and should define all your project dependencies along with versions.
4. You CAN'T have  sections on any other POM file other than parent POM.
5. In each submodule make sure you have Maven dependencies WITHOUT versions.
6. When you introduce a new Maven dependency define it's version under section of parent POM file.  
7. Make sure you have defined following repositories and plugin repositories on parent POM file. These will be used to drag SNAPSHOT versions of other carbon projects which used as dependencies of your project.

Upload a file Using JSP

This demonstration will guide you, how to upload a file into folder (you can define the path) using JSP.

Pre-Request

Application server - I am using Wso2 Application Server. Since it is very user friendly to configure and mange my application using cool user interfaces.

Further Readings,
[1]http://wso2.org/library/articles/2012/10/getting-full-features-apache-tomcat-supported-wso2-application-server
[2] http://docs.wso2.org/wiki/display/AS510/Installation+Guide

Maven - As build tool i am using.

Further Readings,
[1] http://vanjikumaran.blogspot.com/2013/04/installing-maven-and-ant-in-ubuntu.html

let see the codes

Multicast based Clustering With deployment Systematization

In current competitive enterprise environment, integration is playing very important role. Using integration systems are not going to improve the environment, Unless Having right pattern and solution. Having said that, making integration platform in a high availability manner is make more meaningful to scale your enterprise.

In this blog, I am explaining how to make the Carbon 4.2.0 based products in high available manner and how to synchronize the deployment a cross the same products. Furthermore, in this example I have taken WSO2 DSS as product to experiment this feature. You can download this in official website of the WSO2 for free of charge!

Lets have look at the illustrated deployment pattern, I have deployed DSS in 3 diffident machines(Nodes). Important to note is, i have not create Master/Salve setup. Each and every node are in peer to peer way.  All the servers are load balanced by external hardware based "F5 Load Balancer  and inside the Demilitarized Zone(DMZ).

Deployment Diagram

Lets take a step into configuration to understand, How to implement above scenario! 

Step 1 - Downloaded a product into any one of the box and started and shutdown to testing purpose.

Step 2 - Since I have 3 different machine to configure and i wont to use the my domain name instant of localhost/127.0.0.1!!! There for  i used a simple script to replace the values with in "conf" directory.

example - find ./ -type f -exec sed -i 's/localhost/box1.vanji.com/g' {} \;

Step 3 - Configure deployment synchronization in  /repository/conf/carbon.xml

 To understand further on SVN based Deployment synchronization, refer  [1]

Step 4 - Configure registry to maintain the metadata across the all three nodes. Refer further on [2]

Step 5 - Configure User Store to cater external user store. Refer further on [3]

Step 6 - Configure /repository/conf/axis2/axis2.xml to cluster the system.
This step is very important to understand! We are enabling the clustering in multicast mode! Let see the config .

• In Line 2, I have enabled the clustering
• In Line 5, I have mentioned "membershipScheme" as multicast.
• In line 10, I have mentioned "localMemberHost" as name of my current machine "box1.vanji.com"
• And more importantly, i have mentioned my other members detail (Line 20)

Note
I have removed the comment in the configuration to compact the config file!


Step 7 - Just Start the server and observe is there any expedition or error due to the configuration. If there is anything resolve the issues. If there is no miss configuration, You will see the carbon log as above.

 Server is waiting for other nodes to start to make the clustering.
So now stop this sever for while and move to step 8.

Step 8 -  Take the copy of this server and copy to other servers and run the find and replace script to change the name.
Example:- find ./ -type f -exec sed -i 's/box1.vanji.com/box2.vanji.com/g' {} \;

Step 9 - Go to /repository/conf/axis2/axis2.xml and make sure Step 6 followed or not in each and every nodes and make sure Members are defined properly!

Step 10 - Time to Start the servers and enjoy the High Available WSO2 Carbon 4.2.0 based server.

[1] http://docs.wso2.org/display/DSS300/Configuring+the+SVN-Based+Deployment+Synchronizer

[2] http://docs.wso2.org/display/ESB470/Governance+Partition+in+a+Remote+Registry

[3] http://docs.wso2.org/display/DSS310/Configuring+External+User+Stores

ZooKeeper - In my point of view

ZooKeeper: Because coordinating distributed systems is a Zoo

What is Apache Zoo Keeper? 

• Apache ZooKeeper is a software project of the Apache Software Foundation, providing an open source distributed configuration service, synchronization service, and naming registry for large distributed systems.[1]
• Apache ZooKeeper is an effort to develop and maintain an open-source server which enables highly reliable distributed coordination. [2]
• ZooKeeper is a high-performance coordination service for distributed applications. It exposes common services - such as naming, configuration management, synchronization, and group services - in a simple interface so you don't have to write them from scratch. You can use it off-the-shelf to implement consensus, group management, leader election, and presence protocols. And you can build on it for your own, specific needs.[3]

Inherent Problems in Distributed Systems

Little things become complicated because of the distributed nature. For example, when running an application on a local machine, changing of an application involves, editing a configuration file and restarting the app. However, distributed applications run on different machines and need to see configuration changes and react to them. To make matters worse, machines may be temporarily down or partitioned from the network. Robust distributed applications also have the ability to incorporate new machines or decommission machines on the fly. This makes configuration of the distributed application should also be dynamic. 

Distributed applications need a service that they can just believe to oversee the distributed environment. 

The service needs to be as simple as possible and easy to understand as possible. A developer should not have trouble to integrating the service into their application.

The service needs to have good performance so that applications can use the service extensively. 

ZooKeeper aims to meet above requirements by collecting the essence of these different services into a very simple interface to a centralized coordination service. The service itself is distributed and highly reliable. Consensus, group management, leader election and presence protocols will be implemented by the service so that the applications do not need to implement them on their own. 

ZooKeeper instructions show how this simple service can be used to build much more powerful abstractions.

Originally ZooKeeper was a sub project of Hadoop. But from January 2011 Hadoop's ZooKeeper subproject has graduated to become a top-level Apache project. ZooKeeper have Java and C interfaces, and someday they hope to get Python, Perl, and REST interfaces for building applications and management interfaces.

ZooKeeper FileSystem / Data Model

ZooKeeper allows distributed processes to coordinate with each other through a shared hierarchical name space of data registers (we call these registers znodes), much like a tree based file system. Every znode in ZooKeeper's name space is identified by a path. Sequence of path elements separated by a slash ("/"). And every znode has a parent whose path is a prefix of the znode with one less element; the exception to this rule is root ("/") which has no parent. Also, a znode cannot be deleted if it has any children. There are no renames, no append semantics and no partial read writes. 

Data is read and written entirely

The main differences between ZooKeeper and standard file systems are that every znode can have data associated with it (every file can also be a directory and vice-versa) and znodes are limited to the amount of data that they can have. ZooKeeper was designed to store coordination data: status information, configuration, location information, etc. This kind of meta-information is usually measured in kilobytes, if not bytes. ZooKeeper has a built-in sanity check of 1M, to prevent it from being used as a large data store.

Znodes maintain a stat structure that includes version numbers for data changes, ACL changes, and timestamps, to allow cache validations and coordinated updates. Each time a znode's data changes, the version number increases. For instance, whenever a client retrieves data it also receives the version of the data.

1. The data stored at each znode in a namespace is read and written atomically.

2. Each node has an Access Control List (ACL) that restricts who can do what.

3. ZooKeeper supports the concept of watches. Clients can set a watch on a znodes. A watch will be triggered and removed when the znode changes. When a watch is triggered the client receives a packet saying that the znode has changed. And if the connection between the client and one of the Zoo Keeper servers is broken, the client will receive a local notification

ZooKeeper Distributed Architecture

Main Facts

• All servers store a copy of the data in memory 
• The leader is elected at startup 
• Followers respond to clients 
• All updates go through the leader 
• Responses are sent when a majority of servers have persisted the change

The ZooKeeper service itself is replicated over a set of machines that comprise the service. These machines maintain an in-memory image of the data tree along with a transaction logs and snapshots in a persistent store. Because the data is kept in-memory ZooKeeper is able to get very high throughput and low latency numbers. The downside to an in memory database is that the size of the database that ZooKeeper can manage is limited by memory. This limitation is further reason to keep the amount of data stored in znodes small.

The servers that make up the ZooKeeper service must all know about each other. As long as a majority of the servers are available the ZooKeeper service will be available. Clients must also know the list of servers. The clients create a handle to the ZooKeeper service using this list of servers.

Clients only connect to a single ZooKeeper server. The client maintains a TCP connection through which it sends requests, gets responses, gets watch events, and sends heart beats. If the TCP connection to the server breaks, the client will connect to a different server. When a client first connects to the ZooKeeper service, the first ZooKeeper server will setup a session for the client. If the client needs to connect to another server, this session will get reestablished with the new server.

Given a cluster of Zookeeper servers, only one acts as a leader, whose role is to accept and coordinate all writes (via a quorum). All other servers are called followers who read-only replicas of the master. Read requests sent by a ZooKeeper client are processed locally at the ZooKeeper server to which the client is connected. If the read request registers a watch on a znode, that watch is also tracked locally at the ZooKeeper server. Write requests are forwarded to the leader and write requests go through consensus before a response is generated. The rest of the ZooKeeper servers (followers) receive message proposals from the leader and agree upon message delivery. Since followers are replicas of the leader, if the leader goes down, any other server can pick up the slack and immediately continue serving requests. 

The messaging layer takes care of replacing leaders on failures and syncing followers with leaders. Sync requests are also forwarded to another server, but does not actually go through consensus. Thus, the throughput of read requests scales with the number of servers and the throughput of write requests decreases with the number of servers. 

Order is very important to ZooKeeper. (They tend to be a bit obsessive compulsive.) All updates are totally ordered. ZooKeeper actually stamps each update with a number that reflects this order. We call this number the zxid (ZooKeeper Transaction Id). Each update will have a unique zxid. Reads (and watches) are ordered with respect to updates. Read responses will be stamped with the last zxid processed by the server that services the read.

ZooKeeper supports the concept of watches. Clients can set a watch on a znodes. A watch will be triggered and removed when the znode changes. When a watch is triggered the client receives a packet saying that the znode has changed. And if the connection between the client and one of the Zoo Keeper servers is broken, the client will receive a local notification.

ZooKeeper uses a custom atomic messaging protocol. Since the messaging layer is atomic, ZooKeeper can guarantee that the local replicas never diverge. When the leader receives a write request, it calculates what the state of the system is when the write is to be applied and transforms this into a transaction that captures this new state. 

And last but not least, what if you wanted to create a node, which only existed for the lifetime of your connection to Zookeper? That's what "ephemeral nodes" are for .Now, put all of these things together, and you have a powerful toolkit to solve many problems in distributed computing. Zookeeper guarantees completely ordered updates, data versioning; conditional updates (CAS), as well as, advanced features such as "ephemeral nodes", "generated names", and an async notification ("watch") API. 

Benefits

ZooKeeper is Simple, Replicated, Ordered and Fast. ZooKeeper znode provides its clients high throughput, low latency, highly available, strictly ordered access to the znodes. 

1. High Availability ,ZooKeeper's architecture supports high-availability through redundant services. The clients can thus ask another ZooKeeper master if the first fails to answer. ZooKeeper nodes store their data in a hierarchical name space, much like a file system or a trie data-structure. Clients can read and write from/to the nodes and in this way have a shared configuration service. Updates are totally ordered.
2. Performance, The performance aspect of ZooKeeper allows it to be used in large distributed systems. 
3. Reliability, The reliability aspects prevent it from becoming the single point of failure in big systems. Its strict ordering allows sophisticated synchronization primitives to be implemented at the client.

Applications

ZooKeeper is used by companies including WSO2 ,Rackspace and Yahoo! as well as open source enterprise search systems like Solr. Cloudera Inc. Hortonworks Inc. are some other organizations that use ZooKeeper. The Katta project, describes itself as Lucene in the Cloud, a scalable, fault-tolerant, distributed indexing system capable of serving large replicated Lucene indexes at high loads.[8]

References

[1]http://en.wikipedia.org/wiki/Apache_ZooKeeper

[2]http://zookeeper.apache.org/

[3]https://cwiki.apache.org/confluence/display/ZOOKEEPER/Index

[4]http://highscalability.com/blog/2008/7/15/zookeeper-a-reliable-scalable-distributed-coordination-syste.html

[5]http://developer.yahoo.com/blogs/hadoop/posts/2009/03/apache_zookeeper_the_making_of/

[6]http://tech.federatedmedia.net/apache-zookeeper-part-1-convenient-service-servers/

[7]http://www.igvita.com/2010/04/30/distributed-coordination-with-zookeeper/

[8]http://katta.sourceforge.net/

JMS Messagen Store and JMS Message Processor Behavior - WSO2 ESB - Part 3

So why did we meet the EVIL in the my last blog post? Why? yes that is the main reason behind this blog series!!

Now we have to take a deep breath before we dive into the WSO2 ESB and see why this has happened!!

If you carefully study the illustration shown in Figure 1, You will understand whats goes wrong ;)

Figure 1

Anyway, let me take some more time to explain little bit further.
Client sends message to Proxy. "Message Store" persist the message in the message store(JMS Queue). "Message Store" does not persist the message as it is in the JMS Queue. it serializes the message and other information into java serialized object and put it into JMS Queue.When "Message Processor" processes the message, it pulls Messages from JMS Queue and deserializes the java serialized object to further process.

So why did we encounter some exception and other problems in issue 1?
When Message Processor Pull the message from JMS Queue and try to deserialize, Deserilzation process fails since the fetched message is not Serialized by Message Store.

So is that possible to put message into Message Stores's "JMS Queue" ?? NO it is not possible!!!


So whats wrong with Issue 2?
Different JMS listener dose not know how to deserialize the message that fetched from JMS Queue(Message Store)!

Conclusion   

Messages that are put into "JMS Queue" by "Message Store" are can be read only by "Message Processor" and "Message Processor" only capable of understanding the messages that are put by "Message Store".





  

JMS Message Store and JMS Message Processor Behavior - WSO2 ESB - Part 1

I have seen lot of users of WSO2 ESB misunderstood the usage of "Message Stores" and "Message Processor" of the WSO2 ESB. In this Blog series, I will explain further about "Under the Hood" functionalists of the "Message Stores" and "Message Processor" to clear the doubts and myths behind those.

Lets take a very simple use case that i have illustrated bellow,

End of the day, Client is sending a message to Backend in this scenario. Nevertheless, under the hood we are persisting message in JMS queue using "Message Store" and sending it to backend using "Message Processor". This will make sure, we never loss any transaction even backend is fail.

When backend is up and running "Message Processor" send the persisted messages to Backend.

Lets try this concept using WSO2 ESB and ActiveMQ as a message broker. Please refer [1] to understand how to configure WSO2 ESB and ActiveMQ. 

As a Backend and Client, I am using WSO2 ESB SimpleStockQuoteService and Axis2 Client. Please note that, these are packed with WSO2 ESB by default. You can find more detail in [2].

Please refer to the Synapse configuration that has been given bellow which demonstrates the concept above. 

 Use the ant command as given bellow in order to invoke the proxy from client.

ant stockquote -Daddurl=http://localhost:9000/services/SimpleStockQuoteService -Dtrpurl=http://localhost:8280/services/MessageReciveProxy -Dmode=placeorder


As result you, can notice in the Backend, there will be transaction has happened. So underneath, our "MessageReciveProxy"  Proxy received the message and persisted in "JMSMessageStore"  Message Store. Thereafter, "SampleMessageForwardingProcessor"  Message Processor took the message from "JMSMessageStore" and send it to Backend for further process.


So far so good! Hmmmm

In my next post I will introduce the evil ;)





[1]http://docs.wso2.org/wiki/display/ESB470/Configure+with+ActiveMQ
[2]http://docs.wso2.org/wiki/display/ESB470/ESB+Samples+Setup#ESBSamplesSetup-StartingSampleBack-EndServices