So many DBAs; yet so many performance problems

There are many applications and databases around, performing and scaling just as supposed to. We very seldom hear about these project. Of course, this could be small applications with small loads; or, on the other hand, this could be well design application with well written code, and well maintained databases. Yet there are so many systems facing performance and scalability issues. So – what’s this thing about the Oracle DBA’s? First of all – I do not want to blame the Oracle DBAs for really anything, it’s more a scream for assistance. So what do I mean then by saying “So many DBAs; yet so many performance problems”. Well let me try to rephrase the title a little: “So many Oracle DBAs; yet so few Oracle developers”.

What do I mean by this? First. Let me tell you a little about myself. About 10 years ago I used to address myself as an Oracle DBA. Between year 2000 and 2005 I worked as an Oracle consultant, doing Oracle related work for different companies in Norway. This was mainly general Oracle DBA tasks as installing, patching, upgrading, setting up backup routines, support and assistance regarding recovery, monitoring, security, daily maintenance and quite a few performance issues. When facing performance issues we sometimes fixed the problems by creating a new index, changing an index, adding some kind of optimizer hint, pinning something in cache, etc etc. Generally I would say that almost every performance issue I worked with was pointing towards the application. Well … I was pointing towards the application developers, blaming the developers for doing things the wrong way.

Then one day I was hired to assist a development team creating an application using Oracle as the backend data store. Suddenly there were one very demanding project leader and 5-6 developers looking at me, waiting for answers and guidelines regarding how to develop against Oracle. In front of the project startup, I really thought I was an experienced Oracle DBA, surely being qualified for the job. In addition to the experience from practical work, I had been teaching the Oracle DBA courses through Oracle University for several years. Honestly, I think this was the worst job I ever did. Even with an “experienced” Oracle DBA at hand, the project failed when it came to performance and scalability. The data model introduced by the developers was very generic, and by itself a reason to expect a disaster. The list of errors being made in the early phases is embarrassingly long. So … what is the lesson to learn? What do I mean by saying “So many Oracle DBAs; yet so many performance problems”?

The main message is bilateral. First of all. In every development project that is working against an Oracle database, it’s important to involve the necessary Oracle competence as early as possible in the process. Today I work in an development department as an Oracle developer, and I guide our design teams and our Java developers regarding design patterns and coding against Oracle. This way we avoid the big mistakes that definitely will prevent our application from scaling. I also work in a lot of projects where the application is developed and set into production, and now facing performance issues. Most often these issues are caused by lack of Oracle knowledge in the design and development process. Very often these projects have completed without any Oracle knowledge what so ever. The developers know SQL (or they think they know SQL), and the project leaders thing that this knowledge is sufficient. Well – it’s not. Every project, coding against an Oracle database, needs to be – at least – guided by a Oracle developer.

Sometimes I also see that the architects or design team have made some general design decisions, that might drive projects towards disaster. Database independence is one of these design issues that is very common. This might work for a very small application where simultaneity is no issue, but with enterprise solutions there is no such thing as database independence. The databases work in different ways, and should be treated such as. For instance read consistency is handled very differently in different database systems. Also I find that many architects are former developers and tend to prefer doing as much as possible in the application code, for instance – the Java code. This way many design solutions involve avoiding the use of PL/SQL; and on the other side, very often involving APIs such as Hibernate to generate the SQL code. I’ve seen Hibernate been used in very efficient ways, but I’ve also seen the opposite. Hibernate contribute to fast coding and easy mapping between the relational and the object models, but not knowing how Oracle works could easily turn these APIs into causing performance and scalability issues. For instance this could be related to the use of IN-list searches, the use of sequences, and a lot of row-by-row processing. Standards and standardizations is great, but must be used together with knowledge and expedience.

Finally – what about the Oracle DBAs? There are so many Oracle DBAs, but so few Oracle developers. Of course, we need the Oracle DBAs, but we also need Oracle developers to assist in all the different projects going on around the world. The architects, project leaders and developers need our help avoiding doing the big mistakes that we so often see. The problem is that I don’t think they realize it themselves – yet! Some do; and they’re the ones that you never hear about. They represent the projects that perform and scale just as planned and expected.

After contributing to a disastrous project as an Oracle DBA, I realized that my Oracle knowledge had some serious holes. Very important holes. I changed my focus towards the development process. The first approach was reading some great books on the topic (that really should be a must for every Oracle DBA or Oracle developer). This included books written by Tom Kyte, Jonathan Lewis, Christian Antognini, Christopher Lawson and some books from the OakTable. I really recommend this approach towards becoming an Oracle developer. The work is really exciting and challenging, and the need for this knowledge around is obvious. After many years work inside my company, I’ve finally managed to spread the awareness of this necessity. I think this is one of the most important challenges for Oracle DBAs in the future. Don’t tolerate that your company does the same big mistakes as so many others have done before. Let the project leaders, architects and developers in your company know you exist, and that your knowledge could really be of enormous value for their projects. Don’t just be an Oracle DBA, become an Oracle developer!

Post a Comment

Your email is never published nor shared. Required fields are marked *