After seeing some relative success in our Solr implementations xml response times by switching on Tomcats http gzip compression, I've been doing some comparisons between the other formats solr can return. We use Solrnet, an excellent open source .NET Solr client. At the moment, it only supports xml responses, but every request sends the "Accept-encoding:gzip" header as standard, so all you have to do is switch it on on your server and you've got some nicely compressed responses. There is talk of supporting javabin de-serialisation, but it's not there yet. I've decided to compare the following using curl with 1000 rows and 10000 rows in json, javabin, json/gzip compressed and javabin/gzip compressed.

My test setup is a solr 1.4 instance with around 11000 records in sitting behind an nginx reverse proxy handling the gzip compression. As I said, this could easily be achieved by switching on gzip compression in Apache Tomcat. The same 10000 records, returned using the q=*:* directive with wt=json when http gzip compressed is the smallest, but only marginally, compared to wt=javabin. It would seem that json compresses very well indeed. You can also see the massive drop just switching on gzip compression gives to xml.

My conclusion to this would be that because json is a widely accepted content-type, with many well known and fast de-serialising libraries, it would probably be worth implementing that rather than trying to de-serialise javabin. But this was only a quick test and does't take into account how quickly solr handles serialisation of the documents server-side.

Tag: 
Solr
sharri.morris@7digital.com
Thursday, May 8, 2014 - 17:28

Astro Malaysia held it’s annual GoInnovate Challenge Hackathon on the 10th-12th October at the Malaysian Global Innovation & Creativity Centre (MaGIC).

Hopefuls from all over Malaysia massed together for an exciting challenge set by Astro - to build a radio streaming demo. The demo product was meant to redefine the way we watch, read, listen and play with content in two unique hacks to be completed within a 48 hour deadline. Astro offered substantial rewards to those whose ideas that came out on top!

Day 0: Demo - Friday evening

Attendees ranged from junior developers to start-up teams, so long as you’re 18 years old, you can take part!

To begin the Hackathon, entrants were fully briefed and given access to the APIs of both 7digital and music metadata company, Gracenote.

7digital’s lead API developer, Marco Bettiolo, flew in to act as Tech Support for the hackathon.

This photo shows Marco presenting a demo of a radio style streaming service he had previously built.

Day 1: Get Building!

According to the brief, hackers had to choose one of two innovative challenges:

sharri.morris@7digital.com
Tuesday, May 6, 2014 - 17:43

Managing session lifecycle is reasonably simple in a web application, with a myriad of ways to implement session-per-request. But when it comes to desktop apps, or Windows services, things are a lot less clear cut.

Our first attempt used NHibernate's "contextual sessions": when we needed a session we opened a new one, bound it to the current thread, did some work, and unbound the session.

We accomplished this with some PostSharp (an AOP framework) magic. A TransactionAttribute would open the session and start a transaction before the method was called, commit the transaction (or rollback if an exception had occurred), and dispose of the session after the method had completed.

It was a neat solution, and it was very easy to slap the attribute on a method and hey presto - instant session! On the other hand it was difficult to test, and to comprehend (if you weren't involved in the first place), and to avoid long transactions we found ourselves re-attaching objects to new sessions.

These concerns made us feel there was a better solution out there, and the next couple of projects provided some inspiration.

sharri.morris@7digital.com
Thursday, August 8, 2013 - 16:04

Last year we published data on the productivity of our development team at 7digital, which you can read about here.

We've completed the productivity report for this year and would again like to share this with you. We've now been collecting data from teams for over 4 years with just under 4,000 data points collected over that time. This report is from April 2012 to April 2013.

New to this year is data on the historical team size (from January 2010), which has allowed us to look at the ratio of items completed to the size of the team and how the team size compares to productivity. There's also some analysis of long term trends over the entire 4 years.

In general the statistics are very positive and show significant improvements in all measurements against the last reported period:

sharri.morris@7digital.com
Friday, July 19, 2013 - 14:55

Blue and green servers. What?

As part of the 7digital web team's automated deployment process, we now have “Blue-green servers” It took a while to do, but it's great for continuously delivering software.

This system is also known as “red/black deployments” but we preferred the blue-green name as “red” might suggest an error or fault state. You could pick any two colours that you like.

How it works is that we have two banks of web servers – the green servers, and the blue servers. Other than the server names, they’re the same. Only one of these banks is live at any one time, but we could put both live if extra-ordinary load called for it. A new version of the site is deployed to the non-live bank, and then “going live” with the new version consists of flipping a setting on the load balancer to make the non-live bank live and vice-versa.

Why?

Why did we do this? Mostly for the speed. The previous process of deploying a new site version was getting longer. The deployment script would start with a server, upload a new version of the site to it, unpack the new website files, stop the existing web site, configure the new website and start it. Then move on to the next server and do the same.