Blog

All Blog Posts by Mihail Vukadinoff


  • Performance tuning of a fully automated AWS environment started only on schedule

    Performance tuning of a fully automated AWS environment started only on schedule

    We’ve been contacted to conduct a tuning for performance for a huge AWS environment. It is used to host the server side of a mobile application for a TV show that gives the ability to its viewers to vote on various questions during the show. Since the show is scheduled only once per week, it’s a perfect use case for a cloud on-demand environment, that is only being raised during the show, then after it ends, all VMS are shutdown or destroyed. In this way costs are cut to the their minimum, no need to keep expensive private physical servers at all times in a private data center for this.  You can imagine the load from the full set of viewers who are most of the times between 200 000 and 300 000 and they need to vote in parallel during one minute for a certain question. This could be a serious challenge for the system.

    READ MORE
  • Openstack NFS Backend Causing Total Hangs

    Openstack NFS Backend Causing Total Hangs

    I'm not a big NFS fan ever since I worked as a Linux/Unix administrator way back in the good old days. Sometimes when the NFS server hung, lost network connectivity or something else happened all clients that had mounts from the NFS completely blocked waiting for it to come back up, because it is so deep in the kernel. Аll commands, even "ls", froze and the only cure was forcibly rebooting them to get them back online. Neat, eh?When NFS v4.1 emerged, back in 2010 hopes were that it will fix everything. I was a bit sceptical but decided to give it a shot and true, many new fixes in the protocol and implementation were made that enhanced the stability. Some of them were: blocking locks that allow client to ping the server if the lock is released, not only wait for notifications, timeout for server unavailable, parallel access. From what I saw, I couldn’t really break it beyond repair.As time went by Openstack offered the option to have NFS as a storage backend. We decided to use it for one deployment where we saw this technology as appropriate, because we didn't need highly available storage with replication that occupies twice the space, but we needed Cinder volumes to get mounted across the hypervisors. I had a feeling that something could go wrong while making the installation, because I remembered all those nights rebooting servers from ILo / IMPI.

    READ MORE
  • Backing up your virtual machines in Openstack

    Backing up your virtual machines in Openstack

    Backup is an essential part of the IT infrastructure management. Having HA solutions, RAIDs etc. doesn't free you from the need of backup. In case of a human error all those techniques will not save you, only the backup will. However as the saying goes "Your backups are only as good as your restores", so we have to think about regularly checking our backups for consistency.In Openstack it's highly recommended to use Cinder as the main storage provider. Cinder gives you the possibility to create block volumes and attach them to your virtual machines. The best practice is that you keep all your application data onto volumes and not on the instance disk, this disk should be used for the operation system files only, that come from the OS image ( of course packages installed from repositories will also go there) . In this article we will show you more reasons to do so.What you would typically want from a good backup solution is: online backup possibility, easy restores, consistency, easy management, to use as less space as possible.Although it's possible to have a traditional backup solution installed on every virtual machine, Openstack offers us other options to backup our data using snapshotting. The downside is that you can't have an "incremental" snapshot copy yet, you have to store the full size of your snapshots every time you backup. However the simplicity of backups and more importantly restores is far greater than supporting a "in-VM" backup solution that supports incremental backups. 

    READ MORE
  • Oracle Database – Monitor with nagios using check oracle health

    Oracle Database – Monitor with nagios using check oracle health

    This tutorial explains how to set-up the check_oracle_health script (credits to Gerhard Lausser) to work on your Nagios environment on CentOS (or any RedHat based Linux). This nagios plugin allows to monitor many oracle DB parameters – like tablespaces size, session, process count, SGA pool etc. Check it out on the author’s webset.The hardest part of the setup is installing the dependent perl libraries and making modifications in the perl code for them to work.On oracle server we need to create the monitoring user and grant rights, only the minimum necessary for the script to work.

    READ MORE
  • Configuring multiple block storage backends in OpenStack Cinder

    Configuring multiple block storage backends in OpenStack Cinder

    If you're an administrator of virtualized environments you have definetly ran into IO performance issues. IO is the first bottleneck that one hits. Luckily persistant storage has evolved troughout the years and lately we see the high performant SSDs at a reasonable price, still far higher to allow organizations to fully migrate to SSD. The hybrid environments become more and more popular as they combine the low costs for traditional HDD with the high performant Solid state drives. One of the key features of the Cinder storage back- end component of Openstack is the flexibility that allows us to have more than one storage backend on our storage node. This gives us the flexibility to diferentiate the IO heavy applications from the more compute-oriented ones that are more heavy on cpu usage. Typical example is to configure a database VM to run from a SSD drive and the application server cluster to be on a normal storage that is heavily read only during startup. Here is how to achive that with OpenStack and Cinder.

    READ MORE