HornetQ 2.0.0.CR2 is released!

This is a candidate release for the forthcoming 2.0.0.GA release (which is due very soon). You can download it from the web site. :)

HornetQ 2.0.0.CR1 is released!

This is a candidate release for the forthcoming 2.0.0.GA release (which is due very soon). You can download this from the web site. :)

Understanding Connectors & Acceptors

Connectors and accepors are concepts which often confuses new HornetQ users. Both connectors and acceptors are defined in HornetQ server configuration (hornetq-configuration.xml) but users are often confused about when and why they need to configure them. They are described in the user manual but I have a few drawings which could help the users understand them better.

An acceptor defines which type of connection are accepted by the HornetQ server.

A connector defines how to connect to a HornetQ server. The connector is used by a HornetQ client.

HornetQ defines 2 types of acceptor/connector

• invm – this type can be used when both HornetQ client and server run in the same Virtual Machine (invm for Intra Virtual Machine)
• netty – this type must be used when HornetQ client and server runs in different Virtual Machines (this connector type uses the netty project to handle the IO)

To communicate, a HornetQ client must use a connector compatible with the server's acceptor.

You can connect from a netty connector to a netty acceptor (if they are configured with the same host and port):

You can not connect from a invm connector to a netty acceptor:

You can not connect from a netty connector to a invm acceptor:

You can not connect from a netty connector on port 5445 to a netty acceptor on port 5446:

By default netty acceptors and connectors uses localhost as the server address. If the HornetQ client is not on the same machine than the server, it will not be able to connect to it.

One source of confusion is that HornetQ connectors are configured on the server. But I wrote that connectors are used by HornetQ clients, not servers! Why should I configure connectors on the server?

There are two reasons to configure connectors in the server:

1. you want to use JMS & JNDI
2. you want to communicate between HornetQ servers

Using JMS and JNDI

The standard way to use JMS is to lookup JMS resources (ConnectionFactory and Destination) from JNDI.

Context ctx = new InitialContext();
ConnectionFactory cf = ctx.lookup("/ConnectionFactory")
Connection = cf.createConnection();
// the client is now connected to the JMS server

The ConnectionFactory defines how the JMS client can connect to the JMS server. With HornetQ, this means that the ConnectionFactory implementation will use a connector to connect to the HornetQ Server.

First of all, we must define a "netty" acceptor (in hornetq-configuration.xmlk) so that clients can connect remotely to the server:

<acceptor name="netty">
   <factory-class>org.hornetq.integration.transports.netty.NettyAcceptorFactory</factory-class>
  <!-- by default will accept connection on localhost on port 5445 -->
</acceptor>

Then, we define a "netty" connector (in hornetq-configuration.xmlk)so that JMS clients will know how to connect to the server:

<connector name="netty">
   <factory-class>org.hornetq.integration.transports.netty.NettyConnectorFactory<
   <!-- by default will connect to localhost on port 5445 -->
</connector>

Final step is to configure the JMS ConnectionFactory (in hornetq-jms.xml) so that when it is looked up from JNDI, it uses the "netty" connector to connect to the server:

<connection-factory name="ConnectionFactory">
   <connector-ref connector-name="netty"/>
   <entries>
      <entry name="/ConnectionFactory"/>
   </entries>
</connection-factory>

When the HornetQ server is started, it looks like this:

In JNDI, the HornetQ server has stored the configuration associated to the netty connector with the "/ConnectionFactory" binding.

When the JMS client will look up "/ConnectionFactory", it will also retrieve the netty connector configuration and use it to create a netty connector to connect to the server:

To sum up: if you use JMS with JNDI, you MUST configure a connector to connect to the server itself.

Communication between HornetQ server

The other case when you need to define connectors is when HornetQ servers must communicate. For example, they use core bridges, JMS bridges, diverts, they are in the same cluster...

The important thing to remember is that when two HornetQ servers communicate, one server acts as the client of the other server. In that case, the server acting as the client of the other server MUST define a connector to connect to the other server.

Let's take the example of a JMS bridge: Server #1 will host a core bridge which takes messages from the "source" queue on Server #0 and forwards them to the "target" queue:

Server #0 configuration

Server #0 is a regular HornetQ server, its setup will looks like the "JMS & JNDI" case:

• a "netty" acceptor to accept connections from remote clients (one of its clients will be the bridge on Server #1)
• a "netty" connector so that clients can connect to it remotely and send messages to the source queue.

Server #1 configuration

Server #1 is a bit more complex. It acts as a HornetQ server with regards to clients consuming from the target queue but it its bridge is acts as a client of Server #0. Its setup requires:

• a "netty" acceptor to accept connections from remote clients.
• a "netty" connector so that clients can connect to it remotely and receives message from the target queue (as explained in the JMS & JNDI case)
• a "source" connector so that the bridge can connect to the Server #0.

Server #1 defines two connectors, "netty" and "source", which serve different purposes: "netty" is used to connect to the server itself (and will be used by its JMS clients) while "source" connector is used to connect to the other server #0 so that the bridge can receive messages from the source queue.

A note on addresses

Both "netty" connectors and acceptors can be configured with a host parameter. However the meaning of this "host" value is not the same for both:

• a connector will connect to a single server. Its host parameter must correspond to one of the server address (e.g. localhost or macbook.local or 192.168.0.10)
• an acceptor can accept connections from one or many addresses. You can specify a single address (localhost or 192.168.0.10), a list of comma-separated addresses (e.g. 192.168.0.10, 10.211.55.2, 127.0.0.01), or 0.0.0.0 to bind to all the host network interfaces.

Conclusion

Connector configuration can be confusing at first glance but it becomes much more clearer when you follow these simple rules:

• If you use JMS with JNDI, you MUST configure a connector to connect to the server itself
• If a HornetQ server must communicate with another server, you MUST define a connector to connect to the other server

Welcome to two more committers

Welcome to two new committers who have earned their commit rights: Kenny MacLeod (skaffman) and Sergej Zizemski (netwater) Welcome!

HornetQ at the London cloud meetup tomorrow

I'm going to be popping down to the London Cloud Meetup tomorrow (Wednesday 14th Oct) and meeting up with Manik Surtani (Infinspan/JBoss Cache) and Mark Proctor (Drools).

Welcome to a new committer

We have a new committer on board! :) Diego Naya joins us from Plug Tree Labs where he is the CEO. Diego is based in Argentina. Welcome to Diego!

Want to become a HornetQ committer?

It's easy - just follow these steps

Back from JBossWorld and on with HornetQ

Finally I seem to be recovering from an extended period of jet lag after returning from JBossWorld/Red Hat summit in Chicago. The conference was a great success - it was the first time JBossWorld joined forces with the Red Hat guys to create a joint conference. The HornetQ presentation was received very well. We had a good attendance and lots of questions at the end. It's just a shame the presentations weren't video'd - otherwise we could have put it in youtube. Next year I think I will bring along my own video camera and tripod and do it myself! So, what's next? Next stop is HornetQ 2.0 GA. I'm guessing we have around 6 to 8 weeks work to get this out. But of course this depends on how many distractions we are getting. And right now there are a lot ;) Onwards to GA...

A HornetQ Simple Example using Maven

Since HornetQ can be run embedded and can be run with no little or no thirdparty dependencies, its easy to create a simple project that runs an embedded server or simple client. The example code can be downloaded from here and can be built and run as follows: To build both the embedded server and client run mvn package from the root directory. To run the server run mvn exec:java from the embedded-server directory. To run the client run mvn exec:java from the client directory. Now lets look more closely at both examples in more detail. embedded server Lets start by looking at the dependencies in the pom.xml found under the embedded-server directory. The dependencies needed are configured as follows:

<dependencies>
  <dependency>
     <groupId>org.hornetq</groupId>
     <artifactId>hornetq-core</artifactId>
     <version>2.0.0.GA</version>
     <scope>compile</scope>
  </dependency>
  <dependency>
     <groupId>org.hornetq</groupId>
     <artifactId>hornetq-jms</artifactId>
     <version>2.0.0.GA</version>
     <scope>compile</scope>
  </dependency>
  <dependency>
     <groupId>org.hornetq</groupId>
     <artifactId>hornetq-logging</artifactId>
     <version>2.0.0.GA</version>
     <scope>compile</scope>
  </dependency>
  <dependency>
     <groupId>org.hornetq</groupId>
     <artifactId>hornetq-transports</artifactId>
     <version>2.0.0.GA</version>
     <scope>compile</scope>
  </dependency>
  <dependency>
     <groupId>org.jboss.netty</groupId>
     <artifactId>netty</artifactId>
     <version>3.1.0.GA</version>
  </dependency>
  <dependency>
     <groupId>org.jboss.javaee</groupId>
     <artifactId>jboss-jms-api</artifactId>
     <version>1.1.0.GA</version>
     <scope>compile</scope>
  </dependency>

These dependencies are what you would need to run a HornetQ JMS server now lets look at the code needed to run an embedded server:

public class EmbeddedServer
{
public static void main(String[] args) throws Exception
{
  try
  {
     FileConfiguration configuration = new FileConfiguration();
     configuration.setConfigurationUrl("hornetq-configuration.xml");
     configuration.start();

     HornetQServer server = HornetQServers.newHornetQServer(configuration);
     JMSServerManager jmsServerManager = new JMSServerManagerImpl(server, "hornetq-jms.xml");
     //if you want to use JNDI, simple inject a context here or don't call this method and make sure the JNDI parameters are set.
     jmsServerManager.setContext(null);
     jmsServerManager.start();
     System.out.println("STARTED::");
  }
  catch (Throwable e)
  {
     System.out.println("FAILED::");
     e.printStackTrace();
  }
}
}

You can configure the server via the hornetq-configuration.xml file and configure any JMS objects via the hornetq-jms.xml file. Of course you can do much more than this with your server but this is a good starting point. Consult the HornetQ documentation for more information of what other features are available. client The client example shows how you can easily create a HornetQ JMS client using minimal jars. The dependencies this time are much smaller than the server and contain client jars that HornetQ provides for lightweight client apps.

   <dependency>
     <groupId>org.hornetq</groupId>
     <artifactId>hornetq-core-client</artifactId>
     <version>2.0.0.GA</version>
     <scope>compile</scope>
  </dependency>
  <dependency>
     <groupId>org.hornetq</groupId>
     <artifactId>hornetq-jms-client</artifactId>
     <version>2.0.0.GA</version>
     <scope>compile</scope>
  </dependency>
  <dependency>
     <groupId>org.hornetq</groupId>
     <artifactId>hornetq-transports</artifactId>
     <version>2.0.0.GA</version>
     <scope>compile</scope>
  </dependency>
  <dependency>
     <groupId>org.jboss.netty</groupId>
     <artifactId>netty</artifactId>
     <version>3.1.0.GA</version>
  </dependency>
  <dependency>
     <groupId>org.jboss.javaee</groupId>
     <artifactId>jboss-jms-api</artifactId>
     <version>1.1.0.GA</version>
     <scope>compile</scope>
  </dependency>

The client code in this case is copied from one of the HornetQ examples, running this client will send a message to the server that was started earlier.

public static void main(String[] args) throws Exception
{
   Connection connection = null;
   try
   {
       // Step 1. Directly instantiate the JMS Queue object.
       Queue queue = HornetQJMSClient.createQueue("exampleQueue");

       // Step 2. Instantiate the TransportConfiguration object which contains the knowledge of what transport to use,
       // The server port etc.

       Map connectionParams = new HashMap();
       connectionParams.put(PORT_PROP_NAME, 5445);

       TransportConfiguration transportConfiguration = new TransportConfiguration(NettyConnectorFactory.class.getName(),
                                                                                  connectionParams);

       // Step 3 Directly instantiate the JMS ConnectionFactory object using that TransportConfiguration
       ConnectionFactory cf = HornetQJMSClient.createConnectionFactory(transportConfiguration);

       // Step 4.Create a JMS Connection
       connection = cf.createConnection();

       // Step 5. Create a JMS Session
       Session session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);

       // Step 6. Create a JMS Message Producer
       MessageProducer producer = session.createProducer(queue);

       // Step 7. Create a Text Message
       TextMessage message = session.createTextMessage("This is a text message");

       System.out.println("Sent message: " + message.getText());

       // Step 8. Send the Message
       producer.send(message);

       // Step 9. Create a JMS Message Consumer
       MessageConsumer messageConsumer = session.createConsumer(queue);

       // Step 10. Start the Connection
       connection.start();

       // Step 11. Receive the message
       TextMessage messageReceived = (TextMessage)messageConsumer.receive(5000);

       System.out.println("Received message: " + messageReceived.getText());
   }
   finally
   {
      if (connection != null)
      {
         connection.close();
      }
   }
}

You can use this template as a starting point for any projects you may want to create that would use HornetQ as a client or as an embedded server.

Persistence on HornetQ

Persistence on HornetQ is really fast. It is so fast that you would think you are sending non-persistent messages by accident.

Instead of using heavy weighted databases that would provide a bunch of stuff we don't need on HornetQ, we used something faster and still as reliable.

We have written our own circular file Journal that uses either Linux libaio or Java NIO.

Linux libaio is a library that works at the kernel level. We submit writes by sending a DMA Buffer (Direct Memory Access) and a callback interface. The kernel will deal directly with the buffer saving copy time between Java and the disk controller. When the disk is done with the write the callback is returned, and we are sure the data is persisted on the disk.

You may ask, what? At the kernel level?! At the controller level?

Yeah.. that' s the beauty of libaio. It provides system calls to the kernel.

BTW: If you are not a geek who loves programming like I do, you could stop reading this post now :-) Since I will dig a little bit on how it works:

The Journal:

The Journal has a set of pre-allocated files. We keep each file size as close as possible to what would fit on a disk cylinder. We have found a value of 10MiB but that could be different in other systems.

The journal is an append only journal. We always append to the current used file of the pre-allocated set. This way we avoid mechanical movements getting most of the performance possible out of the disk controller.

Deletes are taken as appended records. We add a delete record to the bottom of the file.

We also have a reference counting of records, so when the original file is totally clean (all the records deleted), that file is ready for reuse.

And the journal is transactional also. We have a very nice transactional control, where a commit record is only taken into consideration *if* the entire transaction is on the disk. That gives us ACID control.

SequentialFile interface:

We abstract the disk access through that interface. There are two implementations NIO and AIO. You can select what implementation you want through our configuration. (see User's Manual)

NIO:
This is a very fast approach already. We work at file level, avoiding disk movements. If you don't have Linux or libaio installed in your system, we default to this 100% Java implementation.

AIO (linux libaio):

We have written a small JNI layer that "talks" to libaio on Linux. The basic write method in java, has this signature:

write(int position, int size, ByteBuffer directBuffer, AIOCallback callback). (More detatils on the javadoc)

The buffer here is sent directly to a libaio method called aio_write. (look at aio_write man page).

Another thread will be polling events out of libaio. As soon as the data is on the disk the JNI layer will execute the callback method.

Instead of performing syncs on the disk (which is a slow operation), we use a concurrent latch. You could have many more transactions being executed in parallel. Instead of blocking the whole system while one sync is being performed, we just write as usual and wait for the callback. Each thread will use the most of the performance available at the disk controller. Instead of waiting an expensive sync operation, each thread will be waiting the callback when the data is safely stored.

Conclusion:

Persistence on HornetQ is not only fast but it also scales up when several threads are performing transactions.

This is just one of many of other innovations from HornetQ. We are working hard to make a great software. Feel free to contact us on IRC or our user's forum. We would love to get your feedback.

The Hornet Hatches!

This is an exciting day for messaging at JBoss and Red Hat.

After months of preparation, the middleware messaging team are excited to announce the birth of a new project "HornetQ".

What is HornetQ?

HornetQ is an open source project to build a multi-protocol, embeddable, high performance, clustered, asynchronous messaging system. HornetQ is an example of Message Oriented Middleware (MoM)

HornetQ is designed with usability in mind: We've provided an extensive, easy-to-understand user-manual and quick-start guide and we ship with over 65 ready-to-run examples out of the box, demonstrating everything from simple JMS usage to complex clusters of servers and more exotic functionality.

HornetQ is designed with flexibility in mind: It's elegant POJO based design has minimal third party dependencies: Run HornetQ as a stand-alone messaging broker, run it in integrated in your favourite JEE application server, or run it embedded inside your own application. It's up to you.

HornetQ is designed with performance in mind: Our unique ultra-high performance journal provides never seen before persistent messaging performance. Automatically switching into native mode when running on Linux, it uses asynchronous IO to provide persistent messaging rates that can saturate the write throughput of a disk. Our pluggable transport system uses JBoss Netty out of the box to provide superb performance and scalability on the wire.

HornetQ is licensed using the Apache Software License V 2.0. The ASL 2.0 has fewer restrictions on use than the LGPL, thus providing fewer barriers to adoption. We want HornetQ to be used as widely as possible.

HornetQ has a great feature set that you'd expect of any serious messaging broker.

But... What about JBoss Messaging?

During its development over the last couple of years the HornetQ code-base was worked on under the name JBoss Messaging 2.0

We decided to rename it and separate it as an independent project since it differs in a many ways from JBoss Messaging 1.x and we did not want to confuse the two, quite different, systems. The vast majority of the code base of HornetQ is different to the code base of JBoss Messaging 1.x

So, what happens with JBoss Messaging now? JBoss Messaging 1.x continues to be known under the name of JBoss Messaging and the project is now in maintenance mode only, with all new messaging development happening on the HornetQ project.

Let's go!... to the future

The future has a lot to hold.

What about cloud computing?

Messaging is going to be a key service in the cloud, and our goal is for HornetQ to be the messaging provider of choice in the cloud. It's our view that RESTful APIs will eventually be the preferred API style in clouds. With that in mind we'll be working on implementing a RESTful style API for interoperable messaging.

Since interoperability is high on our list we won't just stop with REST. HornetQ will also be implementing AMQP and native STOMP support to make it a truly multi-lingual messaging system.

Get involved. HornetQ needs you!

The future certainly has lots in store, and there is plenty for us to do. So why not get involved?

HornetQ is a community, open source project and we'd love to hear from you if you'd like to get involved in development, documentation or help in some other way. We have a small team so any help would be fantastic. Join us!

Come, hear the scoop!

To get all the details on this exciting news and more and to see how it fits in the with JBoss' and Red Hat's overall middleware strategy, come and see me and others speak at the Red Hat summit / JBoss World 2009 in Chicago on September 1-4.

See you there!

Any questions?

I've put together a FAQ that should answer most of the common questions about HornetQ.

Here are some more links:

Project web site Project wiki Download User manual Quick start guide Follow us on twitter