Wednesday, December 10, 2014

Remove the payload and send POST request to backend via wso2 esb

There are situations where back-end does not need a payload for POST request. However in wso2 ESB has the payload that mediate before the back-end call. In order to drop the message payload you need to perform two tasks in the wso2 ESB.



Example is given below that demonstrate the use case. 

Tuesday, December 9, 2014

Bind WSO2 Mamanement Console to a certain IP Adress

There are lot of Security measures when deploy the WSO2 Server products. One of them is to restrict the accessibility of the server for certain IP addresses.

In WSO2 serves, it is possible to restrict via the the tomcat valve.

Edit
/repository/conf/tomcat/context.xml 


Add
<Valve className="org.apache.catalina.valves.RemoteAddrValve" allow="127\.0\.0\.1"/ >

How to delete the element in the array of json object in Java-script

Recently we had a very basic requirement to delete an key value pare in the json array using Javascript.


Input data

{"invideoPromotion":{"items":[{"id":{"type":"video","videoId":"11","websiteUrl":""},"timing":{"type":"ms","offsetMs":"2222"},"customMessage":"aa","promotedByContentOwner":"vanji"}]}}

Expected business logic

When type of id is video and websiteUrl is null; websiteUrl should be removed from the payload.

Output data

{"items":[{"id":{"type":"video","videoId":"11"},"timing":{"type":"ms","offsetMs":"2222"},"customMessage":"aa","promotedByContentOwner":"vanji"}]}}

In-order to do this you can use delete functionality of the json in javascript.

Example code



Wednesday, November 5, 2014

How to get the tables list in ms sql server that has the data



This query will help to see the data size details in tables of the MS SQL database.

SELECT 
    t.NAME AS TableName,
    s.Name AS SchemaName,
    p.rows AS RowCounts,
    SUM(a.total_pages) * 8 AS TotalSpaceKB, 
    SUM(a.used_pages) * 8 AS UsedSpaceKB, 
    (SUM(a.total_pages) - SUM(a.used_pages)) * 8 AS UnusedSpaceKB
FROM 
    sys.tables t
INNER JOIN      
    sys.indexes i ON t.OBJECT_ID = i.object_id
INNER JOIN 
    sys.partitions p ON i.object_id = p.OBJECT_ID AND i.index_id = p.index_id
INNER JOIN 
    sys.allocation_units a ON p.partition_id = a.container_id
LEFT OUTER JOIN 
    sys.schemas s ON t.schema_id = s.schema_id
WHERE 
    t.NAME NOT LIKE 'dt%' 
    AND t.is_ms_shipped = 0
    AND i.OBJECT_ID > 255 
GROUP BY 
    t.Name, s.Name, p.Rows
ORDER BY 
    t.Name

Retrieving source IP detail in WSO2 ESB and APIM



These properties will retrieve the respective source IP into the properties.


Remote host 

   <property name="client-host" expression="get-property('axis2', 'REMOTE_HOST')" />

Remote address 

   <property name="client-address" expression="get-property('axis2', 'REMOTE_ADDR')" />

X-Forwarded-For

   <property name="xforward-header" expression="$trp:X-Forwarded-For""/> />

Sunday, September 28, 2014

Connecting and monotoring JMX of WSO2 products in EC2 instance

Most of the time current deployments are hosted in EC2 instance as it is very reliable and scalable. When it come to monitoring the WSO2 products via jmx in EC2 instances. You have to add some parameters inorder to connect with it.


add below parameters in /bin/wso2server.sh


    -Dcom.sun.management.jmxremote \
    -Dcom.sun.management.jmxremote.port=XXXX \
    -Dcom.sun.management.jmxremote.ssl=false \
    -Dcom.sun.management.jmxremote.authenticate=false \
    -Djava.rmi.server.hostname=XXX.XXX.XXX.XXX \

Here port should be port that not used any where in the given instance and host-name should be private IP of the EC2 instances.


Access and monitor the server via Jconsole by JMX url connection that has the domain name and the jmx ports define in the carbon.xml  

Tuesday, August 26, 2014

List of "conditional content aware mediators" and "content aware mediators"

I have Listed down set of "conditional content aware mediators" and "content aware mediators"

conditional content aware mediator

fastXSLT
filter
header
log
property
switch

content aware mediators

bean
cache
callout
clone
command
conditional router
dblookup
dbreport
ejb
enrich
event
payloadfactory
script
spring
store
validate
xquery
xslt
iterate


How to log the Content-Type in WSO2 ESB


To identify the Content-Type in the mediation in wso2 esb.
Please refer to configuration given below. I have modified the sample given in the wso2 esb


Tuesday, August 5, 2014

How to search for installed Software in Ubuntu

Very simple way to identify your installed software in  you Ubuntu machine.

List all the installed software


dpkg --get-selections 

Find the specif software


dpkg --get-selections |grep 

Tuesday, July 22, 2014

Check the available Cipher providers and Cipher algorithms in Java Virtual Machine(JVM)

During the penetration test normally the ethical hacker will also evaluate all the aspects of the Java Virtual Machine(JVM). As a part of it they use to check the weak available ciphers out there in JVM.


Therefore, I have create a simple java code to list of all the available ciphers and their providers in the given Java virtual machine. Please find the code below in my Gist


Disabling weak ciphers in JAVA Virtual machine (JAVA) level

There are known vulnerable weak cipher algorithms are out there such as MD2, MD5,  SHA1 and RC4. Having these in the production servers that have the high sensible data may have high security risk.



When you application running based on Apache Tomcat it is possible you to disable it from the removing relevant cipher from catalina-server.xml.

Example

ciphers="SSL_RSA_WITH_RC4_128_MD5,SSL_RSA_WITH_RC4_128_SHA,TLS_RSA_WITH_AES_128_CBC_SHA,TLS_DHE_RSA_WITH_AES_128_CBC_SHA,TLS_DHE_DSS_WITH_AES_128_CBC_SHA,SSL_RSA_WITH_3DES_EDE_CBC_SHA,SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA,SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA"

Let say SSL_RSA_WITH_RC4_128_MD5 has been identified as a vulnerable weak cipher. So that simply you can remove that from the list and restart the server


ciphers="SSL_RSA_WITH_RC4_128_SHA,TLS_RSA_WITH_AES_128_CBC_SHA,TLS_DHE_RSA_WITH_AES_128_CBC_SHA,TLS_DHE_DSS_WITH_AES_128_CBC_SHA,SSL_RSA_WITH_3DES_EDE_CBC_SHA,SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA,SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA"

Lets say your server is out of control of your hand to control the cipher. Simple but efficient solution is to disable that from the JVM level.

Since Java 1.7 there are two additional properties in $JRE_HOME/lib/security/java.security:


jdk.certpath.disabledAlgorithms=MD2

Controls algorithms for certification path building and validation.

jdk.tls.disabledAlgorithms=MD5, SHA1, RC4, RSA keySize < 1024

This JVM-wide algorithm restrictions for SSL/TLS processing will disable the chipers that listed out there. Furthermore, the used notation is quite obvious here!  it's possible to disallow certain algorithms or limit key sizes.

Note that
Both properties are supported in Oracle JRE 7, Open JRE 7  and IBM Java v7


Further Reading



Thursday, July 17, 2014

Known errors and issue while Running ciphertool in WSO2

I have seen several user mistake and issues that cause the error while running ciphertool.sh of WSO2 carbon servers. So based on my previous experience, I have listed down the error that I encounter so far while using the tool and solution for that...


Error set 1


[vanji@vanjiTestMachine bin]# ./ciphertool.sh -Dconfigure
[Please Enter Primary KeyStore Password of Carbon Server : ]
Exception in thread "main" org.wso2.ciphertool.CipherToolException: Error initializing Cipher
        at org.wso2.ciphertool.CipherTool.handleException(CipherTool.java:861)
        at org.wso2.ciphertool.CipherTool.initCipher(CipherTool.java:202)
        at org.wso2.ciphertool.CipherTool.main(CipherTool.java:80)
Caused by: java.security.InvalidKeyException: No installed provider supports this key: (null)
        at javax.crypto.Cipher.chooseProvider(Cipher.java:878)
        at javax.crypto.Cipher.init(Cipher.java:1653)
        at javax.crypto.Cipher.init(Cipher.java:1549)
        at org.wso2.ciphertool.CipherTool.initCipher(CipherTool.java:200)

This error can cause when keyAlias miss match when generating the key-store, Therefore please reconsider to generate right Key-store with the right keyAlias OR change the values in carbon.xml

Error set 2

I have notice flowing IOError read error while working on windows machine

[Please Enter Primary KeyStore Password of Carbon Server : ]
Exception in thread "main" org.wso2.ciphertool.
CipherToolException: IOError read
ing primary key Store details from carbon.xml file
        at org.wso2.ciphertool.CipherTool.handleException(CipherTool.java:861)
        at org.wso2.ciphertool.CipherTool.getPrimaryKeyStoreData(CipherTool.java
:305)
        at org.wso2.ciphertool.CipherTool.initCipher(CipherTool.java:180)
        at org.wso2.ciphertool.CipherTool.main(CipherTool.java:80)
Caused by: java.io.FileNotFoundException: C:\Program Files\Java\jdk1.6.0_16\bin\
repository\conf\carbon.xml (The system cannot find the path specified)
        at java.io.FileInputStream.open(Native Method)
        at java.io.FileInputStream.(FileInputStream.java:106)
        at java.io.FileInputStream.(FileInputStream.java:66)
        at sun.net.www.protocol.file.FileURLConnection.connect(FileURLConnection
.java:70)
        at sun.net.www.protocol.file.FileURLConnection.getInputStream(FileURLCon
nection.java:161)
        at com.sun.org.apache.xerces.internal.impl.XMLEntityManager.setupCurrent
Entity(XMLEntityManager.java:653)
        at com.sun.org.apache.xerces.internal.impl.XMLVersionDetector.determineD
ocVersion(XMLVersionDetector.java:186)
        at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(X
ML11Configuration.java:771)
        at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(X
ML11Configuration.java:737)
        at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.
java:107)
        at com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(DOMParser.
java:225)
        at com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(Doc
umentBuilderImpl.java:283)
        at javax.xml.parsers.DocumentBuilder.parse(DocumentBuilder.java:180)
        at org.wso2.ciphertool.CipherTool.getPrimaryKeyStoreData(CipherTool.java
:289)
        ... 2 more

There is a windows long classpath issue in the script. Please edit the following lines in ciphertool.bat script 

[vanji@vanjiTestMachine bin]$ ./ciphertool.sh -Dconfigure
[Please Enter Primary KeyStore Password of Carbon Server : ]
Exception in thread "main" org.wso2.ciphertool.CipherToolException: Error initializing Cipher
        at org.wso2.ciphertool.CipherTool.handleException(CipherTool.java:861)
        at org.wso2.ciphertool.CipherTool.initCipher(CipherTool.java:202)
        at org.wso2.ciphertool.CipherTool.main(CipherTool.java:80)
Caused by: java.security.InvalidKeyException: Wrong key usage
        at javax.crypto.Cipher.init(Unknown Source)
        at javax.crypto.Cipher.init(Unknown Source)
        at org.wso2.ciphertool.CipherTool.initCipher(CipherTool.java:200)
        ... 1 more

Edit the line from 73 to 77 with the following lines

call ant -buildfile "%CARBON_HOME%\bin\build.xml" -q 
set CARBON_CLASSPATH=.\conf 
FOR %%c in ("%CARBON_HOME%\lib\*.jar") DO set CARBON_CLASSPATH=!CARBON_CLASSPATH!;".\lib\%%~nc%%~xc" 
FOR %%C in ("%CARBON_HOME%\repository\lib\*.jar") DO set CARBON_CLASSPATH=!CARBON_CLASSPATH!;".\repository\lib\%%~nC%%~xC" 



Error Set 3


[vanji@vanjiTestMachine bin]$ ./ciphertool.sh -Dconfigure 
[Please Enter Primary KeyStore Password of Carbon Server : ] 
Exception in thread "main" org.wso2.ciphertool.CipherToolException: Error initializing Cipher 
        at org.wso2.ciphertool.CipherTool.handleException(CipherTool.java:861) 
        at org.wso2.ciphertool.CipherTool.initCipher(CipherTool.java:202) 
        at org.wso2.ciphertool.CipherTool.main(CipherTool.java:80) 
Caused by: java.security.InvalidKeyException: Wrong key usage 
        at javax.crypto.Cipher.init(Unknown Source) 
        at javax.crypto.Cipher.init(Unknown Source) 
        at org.wso2.ciphertool.CipherTool.initCipher(CipherTool.java:200) 
        ... 1 more 

If you are changed the default keystore privided with wso2server new one, make sure you have change all the references for that keystore. You may have to change the entries in following files. 

WSO2Server/reposotory/conf/carbon.xml 
WSO2Server/repository/conf/security/secret-conf.properties 
WSO2Server/repository/conf/sec.policy 
WSO2Server/repository/conf/security/cipher-text.properties 
WSO2Server/repository/conf/tomcat/catalina-server.xml 
WSO2Server/reposotory/conf/axis2/axis2.xml 

Not only the keysore name, make sure you change keypassword, keystore pasword and keyalias according to your keystore.

Error Set 4


[vanji@vanjiTestMachine:~/software/wso2/wso2esb-4.8.0
$ sh bin/ciphertool.sh -Dconfigure
Exception in thread "main" org.wso2.ciphertool.CipherToolException: IOError reading primary key Store details from carbon.xml file 
at org.wso2.ciphertool.CipherTool.handleException(CipherTool.java:861)
at org.wso2.ciphertool.CipherTool.getPrimaryKeyStoreData(CipherTool.java:305)
at org.wso2.ciphertool.CipherTool.initCipher(CipherTool.java:180)
at org.wso2.ciphertool.CipherTool.main(CipherTool.java:80)
Caused by: java.io.FileNotFoundException: /home/vanji/software/wso2/repository/conf/carbon.xml (No such file or directory)
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.(FileInputStream.java:120)
at java.io.FileInputStream.(FileInputStream.java:79)
at sun.net.www.protocol.file.FileURLConnection.connect(FileURLConnection.java:70)
at sun.net.www.protocol.file.FileURLConnection.getInputStream(FileURLConnection.java:161)
at com.sun.org.apache.xerces.internal.impl.XMLEntityManager.setupCurrentEntity(XMLEntityManager.java:651)
at com.sun.org.apache.xerces.internal.impl.XMLVersionDetector.determineDocVersion(XMLVersionDetector.java:186)
at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:772)
at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:737)
at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:119)
at com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(DOMParser.java:232)
at com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(DocumentBuilderImpl.java:284)
at javax.xml.parsers.DocumentBuilder.parse(DocumentBuilder.java:180)
at org.wso2.ciphertool.CipherTool.getPrimaryKeyStoreData(CipherTool.java:289)
... 2 more



When you run the ciphertool.sh from outside the bin folder  you will see this error and this is the limitation of the tool.


I have listed the issue that i have encountered so far, if i found anything new I will keep update this blog-post with my new findings

Tuesday, July 15, 2014

Write the logs into External database in WSO2 Servers

Some time data mining purpose storing the logs in the database important and it is possible to do that with wso2 carbon products as well.

To achieve above task follow the steps that mention below. I have used mysql for demonstrate this task and it is possible to use and other RDBMS for this.

1. If the server is already running, stop the server.

2. Configure the database (say, LOG_DB) and create the following table (LOGS)
CREATE TABLE LOGS( USER_ID VARCHAR(20) NOT NULL, DATED   DATETIME NOT NULL, LOGGER  VARCHAR(50) NOT NULL, LEVEL   VARCHAR(10) NOT NULL,MESSAGE VARCHAR(1000) NOT NULL);
3. Configure the log4j.properties in the /repository/conf/

Since log4j.rootLogger is already defined append “sql” in it as follows.


log4j.rootLogger=ERROR, CARBON_CONSOLE, CARBON_LOGFILE, CARBON_MEMORY, CARBON_SYS_LOG, ERROR_LOGFILE, sql
Add the following,
log4j.appender.sql=org.apache.log4j.jdbc.JDBCAppender
log4j.appender.sql.URL=jdbc:mysql://localhost/LOG_DB
# Set Database Driver
log4j.appender.sql.driver=com.mysql.jdbc.Driver
# Set database user name and password
log4j.appender.sql.user=root
log4j.appender.sql.password=root
# Set the SQL statement to be executed.
log4j.appender.sql.sql=INSERT INTO LOGS VALUES ('%x', now() ,'%C','%p','%m')
# Define the xml layout for file appender
log4j.appender.sql.layout=org.apache.log4j.PatternLayout


4. Download the mysql driver from, http://dev.mysql.com/downloads/connector/j/5.0.html and place the jar (mysql-connector-java-5.1.31-bin) inside /repository/components/lib/

5. Start the server, you will be getting the logs in the LOGS table as well.



Sunday, July 13, 2014

GIT 101 @ WSO2


Git

Git is yet another source code management like SVN, Harvard, Mercurial and So on!

Why GIT?

Why GIT instant of SVN in wso2?
I do not know why! it might be a off site meeting decision taken in the trinco after landing with adventurous flight trip ;)

  • awesome support for automation story
  • Easy to manage
  • No need to worry about backup and other infrastructure issues.
  • User friendly
  • Publicly your code reputation is available.

GIT in WSO2.

WSO2 has two different repository.
  • Main Repository.
    • Main purpose of this repository maintain the unbreakable code repository and actively build for the continuous delivery story incomprated with integrated automation.
  • Development Repository.
    • Development repository is the place teams play around with their active development.
    • wso2-dev is a fork of wso2 repo!

UPDATE

Now this statement invalid as WSO2 has changed it process on Dec/2014

Rules


  1. Developer should not fork wso2 repo.
    1. Technically he/she can but the pull request will not accepted.
    2. If something happen and build breaks! He/She should take the entire responsible and fix the issue and answer the mail thread following the build break :D
  2. Developer should fork respective wso2-dev repo.
    1. He/She can work on the development on her/his forked repo and when he/she feel build won't break he/she need to send the pull request to wso2-dev.
    2. If pull request should be reviewed by respective repo owners and merge.
    3. On the merge, Integration TG builder machine will get triggered and if build pass no problem. If fails, He/She will get a nice e-mail from Jenkins ;) so do not spam or filter it :D. Quickly respective person should take the action to solve it.
  3. When wso2-dev repository in a stable condition, Team lead/Release manager/ Responsible person  has to send a pull request from wso2-dev to wso2.
    1. WSO2 has pre-builder machine to verify the pull request is valid or not.
      1. if build is passed and the person who send a pull request is white listed the pull request will get merged in the main repository.
      2. if build fails, the pull request will be terminated and mail will send to the respective person who send the pull request. So now, respective team has to work out and fix the issue.
      3. Build pass but not in whitelist prebuild mark it a need to reviewed by admin. But ideally admin will close that ticket and ask the person to send the pull request to wso2-dev ;)
      4. If everyting merged peacefully in main repo. Main builder machine aka continuous delivery machine  build it. If it is fail, TEAM need to get into action and fix it.
  4. You do not need to build anything in upstream, ideally everything you need should fetched from the Nexus.
  5. Allways sync with the forked repository

GIT Basics

  1. Fork the respective code base to your git account
  2. git clone github.com/wso2-dev/abc.git
  3. git commit -m “blha blah blah”
  4. git commit -m “Find my code if you can” -a
  5. git add myAwsomeCode.java
  6. git push


Git Beyond the Basics


  • Sych with upstream allways before push the code to your own repository

WSO2 GIT with ESB team


ESB team owns

Nobody else other than in ESB team has the mergeship :P for these code repository. So whenever somebody try to screw our repo, please take a careful look before merge!
The first principle is no one suppose to build anything other than currently working project.

Good to read

[Architecture] Validate & Merge solution for platform projects

Maven Rules in WSO2


Please find POM restructuring guidelines in addition to things we discussed during today's meeting.  

  1. Top level POM file is the 'parent POM' for your project and there is no real requirement to have separate Maven module to host parent POM file.
  2. Eliminate POM files available on 'component' , 'service-stub' and 'features' directories as there is no gain from them instead directly call real Maven modules from parent pom file ( REF - [1] )
  3. You must have a    section on parent POM and should define all your project dependencies along with versions.
  4. You CAN'T have  sections on any other POM file other than parent POM.
  5. In each submodule make sure you have Maven dependencies WITHOUT versions.
  6. When you introduce a new Maven dependency define it's version under section of parent POM file.  
  7. Make sure you have defined following repositories and plugin repositories on parent POM file. These will be used to drag SNAPSHOT versions of other carbon projects which used as dependencies of your project.