After seeing some relative success in our Solr implementations xml response times by switching on Tomcats http gzip compression, I've been doing some comparisons between the other formats solr can return. We use Solrnet, an excellent open source .NET Solr client. At the moment, it only supports xml responses, but every request sends the "Accept-encoding:gzip" header as standard, so all you have to do is switch it on on your server and you've got some nicely compressed responses. There is talk of supporting javabin de-serialisation, but it's not there yet. I've decided to compare the following using curl with 1000 rows and 10000 rows in json, javabin, json/gzip compressed and javabin/gzip compressed.

My test setup is a solr 1.4 instance with around 11000 records in sitting behind an nginx reverse proxy handling the gzip compression. As I said, this could easily be achieved by switching on gzip compression in Apache Tomcat. The same 10000 records, returned using the q=*:* directive with wt=json when http gzip compressed is the smallest, but only marginally, compared to wt=javabin. It would seem that json compresses very well indeed. You can also see the massive drop just switching on gzip compression gives to xml.

My conclusion to this would be that because json is a widely accepted content-type, with many well known and fast de-serialising libraries, it would probably be worth implementing that rather than trying to de-serialise javabin. But this was only a quick test and does't take into account how quickly solr handles serialisation of the documents server-side.

Tag: 
Solr
sharri.morris@7digital.com
Wednesday, February 20, 2013 - 12:53

Somewhere in the 7digital.com web site infrastructure there are classes that override the default controller and view factories (it is an ASP MVC project). Why did we do this? In our opinion, the default project layout is a hindrance to code readability.

The idea is explained by Uncle Bob in his concept of “screaming architecture”.  i.e. if you glance at the program's folder structure, what is the most blatant thing about it, what is it “screaming about”?

If there's a folder full of controllers, and a folder full of views, and another for models, then it's screaming “I am an ASP.Net MVC project! I do ASP MVC things!”. If there's a folder called “Artists” and another called “Genres”, each containing controllers, views and other classes related to that feature, it's instead saying “I am a music catalogue on the web”.

I personally feel that “screaming architecture” is a very poor name for a very good concept. The architecture isn't having a crisis. It's not running around with hair on fire shouting “aaargh!!!”.  Maybe Uncle Bob has more positive associations with the word “screaming”? With his meaning of “screaming”, every architecture is screaming about something, but what is the important thing. 

sharri.morris@7digital.com
Friday, January 4, 2013 - 10:11

 

We’re primarily driven by meeting 7digital’s goals and objectives

  • Everything we do should be driven by clear business goals and objectives. Where they are lacking we should go and find them.
  • We expect business needs to be provided as problems that need solving with clear expectations and measurables without prejudice towards the implementation.

Release Early and Often; Fail Early and LOUDLY!

  • It’s essential we can respond quickly to changing business requirements. The best measure of our effectiveness in doing so is via frequent predictable releases through a steady rhythm of working. Things need to be easy to change (maintainable) and delivered at a sustainable pace.
  • It’s far more preferable to get something in production as soon as possible and develop iteratively based on feedback than to get bogged down in speculative analysis or a fear of not making all the right decisions up front (be that regarding technology choices or requirements).
  • Failures are expected, and welcome. When projects fail, we learn about other routes that might work. When software fails, it tells us about invalid assumptions we’ve made. The earlier and louder the failure, the more valuable that information is.

The best solutions come from everyone working together

sharri.morris@7digital.com
Wednesday, October 17, 2012 - 09:51

Overview

Servicestack is a comprehensive web framework for .NET that allows you to quickly and easily set up a REST web service with very little effort. We already use OpenRasta to achieve this same goal within our stack, so I thought it would be interesting to compare the two and see how quickly I could get something up and running. The thing that most interested me initially about ServiceStack was the fact that it claims out of the box support for Memcached, something we already use extensively to cache DTOs, and Redis, the ubiquitous NoSql namevaluecollection store.

Getting cracking

I set myself the task of creating a basic endpoint for accessing 7digital artist, release and track details. Whilst taking advantage of ServiceStack’s ability to create a listener from a console window so I didn’t have to waste time attempting to set it up via IIS:

sharri.morris@7digital.com
Tuesday, September 25, 2012 - 16:40

Over the last month we've started using ServiceStack for a couple of our api endpoints (go to the full ServiceStack story here) . We're hosting these projects on a Debian Squeeze vm using nginx and Mono. We ran into various problems along the way which we'll explain, but we also managed to achieve some interesting things; here's a summary. Hopefully you'll find this useful.

Nginx

We're using nginx and fastcgi to host the application. This is good from a systems perspective because our applications can run without root privileges. For the communication between mono-fastcgi and nginx, we are using a unix socket file instead of proxying through a local port. This makes configuration much easier, as you map applications to files rather than port numbers, so the convention rules for this are much more straightforward. (Besides, you may be hit by a memory leak if you don't use unix socket files.) Furthermore, using files instead of ports has made our life easier for automated deployments because: