Value of a Developer

We write software to provide business value. That's what we want to do, and that's what we take pride in. We like a challenge, and want to show that we can solve hard problems and be the hero. We are geeks who want to prove that we are good at our craft. That's what motivates us, and that's a desirable thing to have in a developer.

We are not assembly line workers. We do not as a rule clock in and out on a shift rotation. Some days, we work long and hard to prove that we are awesome and can fix that production bug and make all our users happy. We expect a great deal from ourselves, and we thrill at teamwork and all delivering beyond expectations. After working our arses off to deliver the seemingly impossible, we may take a little extra time getting in the next morning, or come back from lunch a little later. We do the impossible for you, and all we want in return is happy users, and a tiny bit of recognition of what the team has achieved.

Don't snap at your development team if a few people are in a little later in the morning every once in a while. Don't scream and yell if some people come back from lunch a little late after a gruelling morning of frantic coding. If we stop delivering, you can get up tight about it. While we continue to achieve the impossible against all odds, then afford us the flexibility to take mental breaks when we are strained, and to have an extra 15 minutes in bed after spending 4 hours the previous evening researching ways to crack the current insoluble problem.

Most of all, we are all individuals. If I mess up, it's me you want to confront. If my team mate messes up, then confront them. When you punish the whole team like naughty school children because of a few late lunches, even though we continue to deliver time and time again, then expect us to be majorly pissed off. When we're pissed off, you can guarantee we will find it harder to solve those impossible problems within those impossible deadlines. You can expect us to find it harder to see the problems as our problems rather than your problems. Give us some respect, because I'm damn sure we've earned it.

SQL for distributed caches

Once again, I find myself on a project that is struggling to get data in and out of the database fast enough.  Given the steps involved in persisting and retrieving objects from the database, it's hardly surprising.  Okay, I know there are better ways to go about solving the cost of this process, but here are the steps:-

1. Use XStream to serialise most of the object data to XML.
2. Take the resulting smaller object, and use Hibernate to map it to a table in the database.
3. Fetch the next number in a hibernate generated HiLo sequence.
4. Store the record, with the XML part in a Blob.
5. Triggers create audit details
6. Re-query the object, to fetch the timestamp audit data

So that's my data persisted.  Obviously there are similar steps involved in retrieving the data.  So what is the purpose of the database?  Well, we want to be able to deal with large volumes of data, without running out of memory.  We want to retain data once the vm stops.  We want the data to be available to all the other servers either as federated services, or as multiple different services with some overlap in data requirements.  We want to be able to report on the information.  We want to be able to query the data to pull out just the results we are interested in.  All the usual reasons.

Given the time critical nature of the system though, performance is the biggest priority.  If you've got to run thousands of simulations across a complex trade portfolio in order to feed back to customers waiting to find out price for the products they have requested, the faster you can provide results, the more business you can accept.  After analysing where the bottlenecks are, as per usual the database is the slow part.  Now, this isn't intended to be a database rant.  When I say 'It's the database thats the bottleneck', I'm actually covering far more than the database.  The whole object marshalling, XML generation, Hibernate overheads, sub-optimal queries generated by a mis-match between a generic tool (hibernate) and a specific database setup, network overheads, poorly indexed or structured schemas ... the list goes on.  It's all persistence layer stuff.

So I want to remove the overhead of all that object relational mapping stuff, but retain the benefits of a relational database.  I want to be able to use external reporting tools which use sql and expect relational data back.  I want to be able to query data and just retrieve the subset I ask for, using all the usual joins and sub-queries.  I just don't want to have to marshal the objects to and from the database.  I don't want the additional layer in my primary data owning system which is causing so much overhead.

Okay, that's the wants, now where's my solution?  From a storing and retrieving data perspective, I really just want a Map which I can throw my objects in and forget about them, and a filter approach for pulling the matching records out again.  Nothing new, strange or unusual about that.  We can do that with Coherence, or Terracotta.  There's a whole heap more, but in my little corner of the world, they are the main ones.  These are great technologies for eliminating the relational part, but they don't support a SQL like approach.

Is there an existing product that allows me to store my objects in a distributed cache, and permits me to query that cache with SQL?  Sort of like a JDBC driver that reads from Terracotta?  I think that would be the ideal solution.  That allows me to have my relational representation of the data, and not have to marshal to and from the DB all the time.

Any suggestions??

A New Year, A New Language

A colleague of mine, Ivan Moore, suggested some years ago that you should as a developer learn a new language every year. Since then, I've tried to follow the idea. This year is the year of Scala, and so far I'm really enjoying it.

I have no prior experience of a functional language, so Scala offers the opportunity to learn something new. I also like the idea that I can compile my Scala components and package them as a jar, which is seamlessly integrated with Java code, meaning that in theory at least, I could create components in my existing project at work!

Challenge your Python

A few days ago I came across the PythonChallenge website. I know I'm late to the party as it's been around for a couple of years. Since then, I've been hooked. It's a series of puzzles to solve, that all involve writing a little script to solve them. It's much better than watching telly as a way of relaxing once the kids are in bed.

If you have a basic exposure to python, and want some exercises to help you learn more, then I would highly recommend it. I'm lovin' it.

Comfort Zone

I like writing code. It's not only something I enjoy, but I like to think I'm quite good at it. I guess it's the fact that it's just me and the machine, tunnel vision, cranking out a solution to a problem.

I took on the extra curricular activity of being a governor at my son's school about 2 years ago. It's been very rewarding finding out how his school ticks. I went on a course about 'Performance Management' which covered what is involved in conducting the Head teacher's performance / pay review. This is proper manager stuff. Trouble was, there was only me and one very new governor that went on the training, so I was therefore the most qualified to lead the review.

I will confess to being more than a little apprehensive. I don't respond well to authority, so actually being the authority made me more than a little uncomfortable. I read through all the notes from past meetings that I could get my hands on, and the various policies and pupil performance metrics etc, but I still felt hopelessly out of my depth as I walked in to the meeting. Fortunately, the Head teacher in question has done a fantastic job, so that was a relief. In the end, the meeting took over 2 hours with me asking questions and examining evidence to assess past performance, before identifying targets for the year ahead. By the time I got home afterwards I was absolutely exhausted.

I learnt a lot from the exercise. I now know that I can do the management type thing if I really have to, and that I should probably have a little more respect for the work that managers do. I know how draining it was doing 1 review ... imagine managing a team of 10 !

Getting a grip

I tried out a small climbing wall whilst on holiday in Center Parcs. It was a 15 minute session, which ended up being 10 minutes after the safety talk, and getting harnessed up. Even so, it was lots of fun. Since getting back from that holiday, I've often thought about trying some more climbing. So I searched for a local climbing wall, and found a place about 12 miles away with loads of climbing walls and boulders.

I was talking to a friend of mine, who went climbing several years ago. He liked the sound of it, so we headed across to the WestWay sports club to try out their 2 hour taster session. It started off really quickly, with a quick demo of how to tie the right knots, before being sent up the wall, with my friend holding on to the other end of the rope, taking up the slack as I climbed. It was awesome fun. I can highly recommend it.

We tried 3 different vertical climbs of anywhere up to 15 metres high. We also tried some bouldering, where you climb without ropes, but you never get more than a few feet off the ground. It sounds easy, but you start off almost as if you're inside a cave, and climb across the ceiling to get out and round. Very cool.

By the time the session had finished, I was exhausted. My grip had completely gone. It took a couple of days after that before I could grip a pen properly! Only problem now is that I can't wait to go back and try it again. Think I'll go for the beginners course next.

Leading from the front

It's been the pattern for a few years now.  Hang around in a team for too long, and I go and get "promoted".  Maybe it's my big mouth that gets me into this kind of trouble, but for some reason, managers seem to think that putting me in a leader role is a good thing.  As seems also to be the pattern, I consistently fail to dodge the position.

This morning, when there was yet another piece of testing to be done in an awkward corner of the code, another member of the team was trying their best to avoid it, so I gladly volunteered to take it on.  When asked why I was so willing, I just automatically replied 'Lead as you would wish others to follow'.  I may be terribly naive, but I think my point hit home.  Sometimes you get to do the cool jobs, like spiking the new shiny libraries, and other times you get given a shovel, and it's time to muck out.