17 sept. 2015

Apache Camel - Unit Testing Spring route

Apache Camel is an awesome framework, also providing a unit testing framework.

I found it quite tough to dive into that testing framework as testing a route is not the same than testing a regular Java class.
Here are some pattern when testing your Spring route :

Base code

The spring route

Assuming this route in camel.xml :

 

 
  
 

 

 
 

Test class skeleton

And this JUnit test class, referencing
  1. The XML route file
  2. The @DirtiesContext allowing us to manipulate the Camel Context on tests without effects outside of each test
   
@ContextConfiguration("classpath:camel.xml")
@DirtiesContext
public class MyTest extends AbstractJUnit4SpringContextTests{

    @Autowired
    private ModelCamelContext context;

    @Test
    public void testRouteId() throws Exception {
       // introspect route
       // create expectations
       // execute route
       // asserts
    }
}

Patterns to test the route

Test a component

Here we are mocking the parameters provided to the WebService :
  1. Create a MockEndpoint
  2. Insert endpoint in the route
  3. Add expectations for the endpoint :
    1. The number of messages
    2. The type of the message
    3. The content of the message
  4. At the end of the test method, trigger the assertion

    @EndpointInject(uri = "mock:assertWsParams")
    protected MockEndpoint assertWsParams;
...

//introspection
context.getRouteDefinition("route-id").adviceWith(context, new AdviceWithRouteBuilder() {
 @Override
 public void configure() throws Exception {
  weaveById("cxfWs").before().to(assertWsParams);
 }
});

//expectations
assertWsParams.expectedMessageCount(1);
assertWsParams.expectedBodyReceived().body().isInstanceOf(RequestType.class);
assertWsParams.expectedMessagesMatches(new Predicate() {
 @Override
 public boolean matches(Exchange exchange) {
  return ((RequestType) exchange.getIn().getBody()).getId().equals(EXPECTED_ID);
 }
});

//execution
...

//assertion
assertWsParams.assertIsSatisfied();

Mock a component

Here we are mocking the cxf WebService with a static response :
  1. Find the "node" with the weaveById
  2. replace() it
  3. with a process()
context.getRouteDefinition("route-id").adviceWith(context, new AdviceWithRouteBuilder() {
 @Override
 public void configure() throws Exception {
  weaveById("cxfWs").replace().process(new Processor() {
   @Override
   public void process(Exchange exchange) throws Exception {
    ReponseType res = new ResponseType();
    res.setCode("200");
    res.setMessage("OK");
    exchange.getOut().setBody(res);
   }
  }).id("cxfWsMock");
 }
});

Run the test

Here we are running the test, injecting some data in the route to test it and the assertions :
  1. Create a producer on a uri
  2. Send some data
@Produce(uri = "direct:hello")
protected ProducerTemplate producer;
...

producer.sendBody("Hello");

Conclusion

My tests are mainly based on these 3 patterns, be sure to understand the logic.

More information on Camel test documentation

12 juil. 2015

Security - Tips to develop a secured Java web application

I recently had to build a web application aiming to handle critical and confidential user data.

Security was the main challenge of this application.

Of course, security is a wide topic, so I 'll present here the security measures I had to take in the code to ensure data privacy and pass the security audit (black box and gray box)

OWASP

The bible of Web Security is the OWASP web site

The measures I had to take to counter the top 10 risk :

Web tiers

  • Detect and reject some patterns on form submission ("<script" for example) (A3)
  • Include a one time token inside every form, and challenge it server side on form submission (A8)
  • Include a one time token inside every link that perform an write operation in database, and challenge it server side on action (A8)
  • Configure and test the HTML escaping of variables of the framework rendering engine. (A3)
  • Develop a full set of features to keep ids from being transfered client side. My strategy was to store ids in a "session context" and rely on it, transfering indexes instead - in the case of tables, for example (A4)
    • Beware, sometimes the magic of your framework car introduce this vulnerability without even noticing!
  • Restrict every entry point for the role allowed - based on a standard RBAC.
  • Initiate redirects server side, redirecting to a static declared URL (no computation based on user input) (A10)
  • Set the pragma:no-cache header

Database tiers

  • 100% names queries. No concatenation. Never. (A1)
  • Escaping of data rendered in PDF report. (A1)

App tiers

  • Check the application server session cookie value algorithm (java.secure.Random) (A2)
  • Set session timeout to an accurate duration (A2)
  • Set the cookie-config to secure in web.xml - https only app (A2)
  • Hide every HTTP response header that could give information to attacker

Other

  • Configure iptables as "deny by default" with rules rejecting connection on unallowed port (A5)
  • Enforce authorization between software components, according to the vendor specification(restrict by IP for example) (A5)
  • Include operations in security configuration (A5) 
  • Detect and protect sensitive data - including project owner in the process (A6)
  • Stay updated on your technologies - mostly version updates and known vulnerabilities (A9)

Conclusion

This is obviously not exhaustive, common sense is important as well as training developers.

Anyway, we passed 3 security audit (code, web security, server vulnerability), not so bad !

28 mai 2015

Ionic : differences between desktop browser and android web view

Ionic is a powerful framework to develop hybrid mobile applications on top of angularjs.

Here is what i learned from my first android tests - based on Ionic 1.0.0.rc5 - simply running ionic run android command

Error : device is not defined

The main issue i met. Everything was correctly defined, including cordova.js, the javascript import order etc

The cause of the issue was actually that I perform a synchronization on application launch, triggered by the first controller - to be accurate.

Controllers are not evaluated once ionicPlatform is ready :(.

So to skip that error, the synchronization has to be wrapped "on platform ready"

$ionicPlatform.ready(function(){
   syncService.sync();
});
 

Full width/height Background invisible

My application displays a beautiful background, unfortunately android displayed nothing.
This is because android doesn't seem to enjoy a full CSS declaration...

Just change the declaration and everything becomes alright.
/* fail */
.scroll-content{
    background: url("../img/background.png") no-repeat scroll center center / 100% 100% transparent
}
/* success */
.scroll-content{
    background-image: url("../img/background.png");
    background-repeat: no-repeat;
    background-size: 100% 100%;
}

Dates gap

The dates displayed by the application are transmitted by the server as ISO 8601, easily converted through moment.js.

Except the timezone in my device was not the same one than my desktop browser.
Be sure to set this correctly.

Ionic application reboot after a camera call

A though error once again : on corodova camera callback, the application reboot and becomes "ko".

This is a configuration of the device. In your device settings, just uncheck the "do not keep activities" configuration. Just do it now.

Clarification on grid system

I had some issues by not clearly declaring the "col" CSS class on "columns".

Instead of :

...
Declare :

...

Tabs are top instead of bottom

Something weird : tabs, that are displayed bottom on desktop, moved to top in android!

Add this to your angularjs .config function  :


$ionicConfigProvider.tabs.position("bottom");//force bottom
$ionicConfigProvider.tabs.style("standard"); //force same look and feel

Header view title not centered !

Similar issue.
Add this to your angularjs .config function  :


$ionicConfigProvider.navBar.alignTitle('center');

4 mai 2015

Spring Boot your Java batch

What I hate with batch development is the packaging phase.
Where Java EE normalizes everything, batches have been totally forgotten.

As often, Spring comes with a solution and makes our life easier with Spring Boot.

A revolution.

It's "magic".

1. Your batch

Develop your batch, as usual. Using Spring Batch framework is 'not' a requirement. It simply eases the integration and the configuration after packaging.
I personnally rely on maven. Just add the following lines in you pom.xml

 org.springframework.boot
 spring-boot-starter-batch
 1.2.3.RELEASE
 
  
   org.springframework.boot
   spring-boot-starter-logging
  
 



 
  
   org.springframework.boot
   spring-boot-maven-plugin
   1.2.3.RELEASE
   
    
     
      repackage
     
    
   
  
 

Running mvn package now create 2 jar :
- .jar.original -> the jar before Spring Boot complete it
- .jar -> the runnable fat jar built by Spring Boot

You can run it using the command :
java -jar myfatjar.jar


2. If your batch is developed with Spring

Name your configuration file application.properties.
No need to load it by Spring configuration. This file name is a "keyword".
More details about this "trick" in pitfall section.

3. Write you "main" class

Just write it the following way.
Notice the Spring loading. You can provide a Configuration Bean or a XML file. Depends on your habits.
@SpringBootApplication
public class HelloApp {
    public static void main(String[] args) {
        SpringApplication.run(new Object[]{HelloApp.class, "spring-boot.xml"}, args);
    }
}

4. Run it with external configuration

Externalizing the configuration is not an option for production, it's a requirement.

Run your batch with the following arguments allow you overwrite any configuration present in your application.properties.

... --spring.config.location=file:conf/file1.properties,file:conf/file2.conf ...

5. And external logging

To externalize the logging (log4j.xml file in this example), this requires a bit of additionnal configuration.
As you may have noticed, spring-boot-starter-logging has been excluded from the pom.xml.
The reason is simple : the default transitive dependency run with slf4j, whereas I work with log4j.

Add this dependency instead :

...

 org.springframework.boot
 spring-boot-starter-log4j
 1.2.3.RELEASE

...

Reference the log4j file as an argument on batch startup command line.

... --logging.config=file:conf/log4j.xml 

6. Final packaging and runtime configuration

Finally, we end up with the following run command line (Windows style) :

cd %~dp0
java -jar myfatjar.jar --spring.config.location=file:conf/file1.properties,file:conf/file2.conf --logging.config=file:conf/log4j.xml


And the following production file structure :

7. Pitfall

I had some issues with overriding property file loaded in my Spring configuration.
The internal jar configuration took precedence over configuration provided as arguments during launch.
I don't think this loading order is logical, I may have missed something... I fell back naming my configuration file application.properties

10 août 2014

Migrating Phaser to CocoonJS

This post to detail my journey on making a Phaser 2.0.7 game work with CocoonJS 2.0.2 (versions ARE importants here).

 

Patch Phaser

Follow this wiki page : https://github.com/photonstorm/phaser/wiki/Phaser-General-Documentation-:-CocoonJS

 

JQuery

So, first of all, JQuery doesn't load  on CocoonJS navigator.
Move on to underscorejs.

 

XML

That may be sound crazy but CocoonJS doesn't provide any XML API. I draw some texts as Phaser bitmapText requiring an XML descriptor parsing.
This code saved my life : https://github.com/videlais/xml-for-cocoonjs

 

Scaling

I never had any issue with scaling during my previous tests (desktop, cordova build on mobile...) but there problems arise : my stage where truncated with black gaps on sides.
In phaser, my game is created using safe height, with the safe width computed based on safewidth/safeheight ratio, then scaling apply

 
this.game = 
    new Phaser.Game(widthRatio, safeHeightheight, 
    Phaser.CANVAS, 'hh_canvas', null, false, false);

hh.game.scale.setScreenSize(true); 
 
With CocoonJs the code is fairly different : full window w&h have to be applied on Game creation, and disabled setScreenSize().

this.game = 
    new Phaser.Game(window.innerWidth*window.devicePixelRatio, window.innerHeight*window.devicePixelRatio, 
    Phaser.CANVAS, 'hh_canvas', null, false, false);   

 

Black background

A fun one that took me a lot of time to figure out the problem.
I render my background using a gradient filled bitmapData : black in CocoonJS
Do NOT set transparent background on new Phaser.Game creation. Works everywhere but in CocoonJS.

 

Last sprite/image not rendered

For an unknown reason, the last element I add to game in "create" function is never renderered.
I just added a dummy.png 1x1 empty image on all my State create() function...

 

Animated texts truncated

One more a time a crazy problem : the last character of my texts, when animated, is not rendered.
Had to add a suffix "_" in case of CocoonJS navigator.
if(navigator.isCocoonJS){
    _text = _text+"_"//add a dummy last character
} 

Hope I saved you some hours I wasted.

I ll try to keep this post updated.

2 août 2014

Grails Offline mode with Maven dependency resolution

Developing with Grails 2.3.5, the maven dependency resolution is almost perfectly integrated.
It depends on the aether core engine to resolve dependencies.

A problem arises when working on offline mode, without any way to check for artifacts repositories.
The application in development runs and gets stuck on "|Configuring classpath" step... for minutes until the application finally starts!
Unfortunately, the configuration grails.offline.mode=true doesn't work with maven resolution, only with the legacy "ivy" dependency resolution.

So here is the trick, after trying loads of configurations failing (-Dsun.net.client.defaultReadTimeout, -Dsun.net.client.defaultConnectTimeout), I ended up adding the following parameters to the JVM :

-Daether.connector.connectTimeout=500 -Daether.connector.requestTimeout=500

Configuring faster timeouts, I can't even notice a latency during application boot :)

Productivity is back !

7 juin 2014

Grails and custom HQL functions and types

Developping database layer with Grails is pretty fun, I like the Closure Criteria notation.

One of my need was to query some Postgresql database columns, columns being crypted. I had to extend GORM Hibernate mapping and query to be able to work with crypting. Grails 2.3.5, Hibernate 4.1.

Mapping Domain property

First job was to map my Domain properties with a custom Hibernate type.


Hibernate UserType :
public class StringEncrypted implements UserType {

    // Here we will manipulate only BINARY persisted types (encrypted data are persisted into binary format)
    static final int SQL_TYPE = Types.BINARY
    static final int[] SQL_TYPES = [ SQL_TYPE ]

    private static boolean initialized = false;

    protected static CipherService cipherService = null;

    /**
     * The class returned by nullSafeGet().
     *
     * @return Class
     */
    @Override
    Class returnedClass() {
        return T.class
    }

    /**
     * Return the SQL type codes for the columns mapped by this type.
     * The codes are defined on java.sql.Types.
     *
     * @return int[] the typecodes
     * @see Types
     */
    @Override
    public final int[] sqlTypes() {
        return (int[]) SQL_TYPES.clone();
    }

    /**
     * Compare two instances of the class mapped by this type for persistence "equality".
     * Equality of the persistent state.
     *
     * @param x
     * @param y
     * @return boolean
     * @throws HibernateException
     */
    @Override
    public final boolean equals(final Object x, final Object y)
            throws HibernateException {
        return x == y || ( x != null && y != null && x.equals( y ) );
    }

    /**
     * Get a hashcode for the instance, consistent with persistence "equality".
     *
     * @param x
     * @return
     * @throws HibernateException
     */
    @Override
    public final int hashCode(final Object x)
            throws HibernateException {
        return x.hashCode();
    }

    /**
     * Retrieve an instance of the mapped class from a JDBC resultset. Implementors should handle possibility of null values.
     *
     * @param rs a JDBC result set
     * @param names the column names
     * @param session
     * @param owner the containing entity
     * @return Object
     * @throws HibernateException
     * @throws SQLException
     */
    @Override
    public Object nullSafeGet(final ResultSet rs, final String[] names,
                              final SessionImplementor session, final Object owner)
            throws HibernateException, SQLException {

        checkInitialization();

        final byte[] message = rs.getBytes(names[0])

        return rs.wasNull() ? null : convertToObject( cipherService.decrypt( message ) )
    }

    /**
     * Write an instance of the mapped class to a prepared statement.
     * Implementors should handle possibility of null values.
     * A multi-column type should be written to parameters starting from index.
     *
     * @param st a JDBC prepared statement
     * @param value the object to write
     * @param index statement parameter index
     * @param session
     * @throws HibernateException
     * @throws SQLException
     */
    @Override
    public void nullSafeSet(final PreparedStatement st, final Object value, final int index,
                            final SessionImplementor session) throws HibernateException, SQLException {

        checkInitialization();
        if (value == null) {
            st.setNull(index, SQL_TYPE);
        } else {
            st.setBytes(index, cipherService.encrypt( convertToByteArray( (T)value ) ) );
        }

    }

    /**
     * Return a deep copy of the persistent state, stopping at entities and at collections.
     * It is not necessary to copy immutable objects, or null values, in which case it is safe to simply return the argument.
     *
     * @param value the object to be cloned, which may be null
     * @return Object a copy
     * @throws HibernateException
     */
    @Override
    public final Object deepCopy(final Object value)
            throws HibernateException {
        return value;
    }

    /**
     * Are objects of this type mutable?
     *
     * @return boolean
     */
    @Override
    public final boolean isMutable() {
        return false;
    }

    /**
     * Transform the object into its cacheable representation. At the very least this method should perform a deep copy if the type is mutable.
     * That may not be enough for some implementations, however; for example, associations must be cached as identifier values. (optional operation)
     *
     * @param value the object to be cached
     * @return a cachable representation of the object
     * @throws HibernateException
     */
    @Override
    public final Serializable disassemble(final Object value)
            throws HibernateException {
        if (value == null) {
            return null;
        }
        return (Serializable) deepCopy(value);
    }

    /**
     * Reconstruct an object from the cacheable representation.
     * At the very least this method should perform a deep copy if the type is mutable. (optional operation)
     *
     * @param cached the object to be cached
     * @param owner the owner of the cached object
     * @return a reconstructed object from the cachable representation
     * @throws HibernateException
     */
    @Override
    public final Object assemble(final Serializable cached, final Object owner)
            throws HibernateException {
        if (cached == null) {
            return null;
        }
        return deepCopy(cached);
    }

    /**
     * During merge, replace the existing (target) value in the entity we are merging to with
     * a new (original) value from the detached entity we are merging.
     * For immutable objects, or null values, it is safe to simply return the first parameter.
     * For mutable objects, it is safe to return a copy of the first parameter.
     * For objects with component values, it might make sense to recursively replace component values.
     *
     * @param original the value from the detached entity being merged
     * @param target the value in the managed entity
     * @param owner
     * @return the value to be merged
     * @throws HibernateException
     */
    @Override
    public final Object replace(final Object original, final Object target, final Object owner)
            throws HibernateException {
        return original;
    }

    /**
     * Charset used for the non-encrypted data
     */
    public static String CHARSET = "UTF-8"

    @Override
    protected String convertToObject(byte[] byteArray) {
        return new String( byteArray, CHARSET )
    }

    @Override
    protected byte[] convertToByteArray(String object) {
        return object.getBytes( CHARSET )
    }

    /**
     * Checks if cipherService is initialized, and initializes it if necessary
     */
    protected static synchronized final void checkInitialization() {
        if (!initialized) {
            cipherService = Holders.getGrailsApplication().mainContext.getBean("cipherService")

            initialized = true;
        }

    }

Then apply this type to any Domain property :
String label
static mapping={
        label type: StringEncrypted
}

From now, the encryption/decryption will be automatically applied on Grails save and find. Magic !

Add decryption function to HQL language

To query with decryption "on the fly" by the databse engine, this requires first that you crypting algorithm is exactly the same between your JVM and your database. Use "AES/CBC/PKCS5Padding" between Java and Postgresql.

In you BootStrap.groovy, add a decrypting function to Hibernate :

def dialect = grailsApplication.mainContext.sessionFactory.dialect

        def decryptString = new SQLFunctionTemplate(BinaryType.INSTANCE, "convert_from(decrypt(?1, convert_to('"+grailsApplication.config.cipherKey+"', 'UTF8'), 'aes'), 'UTF8')"){
            public Type getReturnType(Type columnType, Mapping mapping){
                return StringType.INSTANCE;
            }
        }
        dialect.registerFunction('decryptString', decryptString)

From now, a new HQL function is available : decryptString.

Query

You can now decrypt on the fly in your HQL function :


def res = DemoDomain.executeQuery(
                "FROM DemoDomain \
                WHERE decryptString(label) = :testValue", [testValue : testValue])[0]

Decryption is processed by the database to query on crypted column.

Note

  1. This raises performance issues as the query performs a full table scan... Well sometimes you just haven't the choice but to query this way.
  2. Couldn't find a way to query with add custom function on Closure notation. What a pity.
Thanks to this blog, really help me.
Fourni par Blogger.