Thursday, 21 September 2017

Kernel tuning in Kubernetes

Kubernetes is a great piece of technology, it trivialises things that 5 years ago required solid ops knowledge and makes DevOps attainable for the masses. More and more people start jumping on the Kubernetes bandwagon nowadays, however sooner or later people realise that a pod is not exactly a small VM magically managed by Kubernetes.

Kernel tuning

So you've deployed your app in Kubernetes, everything works great, you just need to scale now. What do you do to scale? The naive answer is: add more pods. Sure, you can do that, but pods will quickly hit kernel limits. One particular kernel parameter I have in mind is net.core.somaxconn. This parameter represents the maximum number of connections that can be queued for acceptance. The default value on Linux is 128, which is rather low:
root@x:/# sysctl -a | grep "net.core.somaxconn"
net.core.somaxconn = 128
You might get away with not increasing it, but I believe it's wasteful to create new pods unless there is a cpu or memory need.

In order to update a sysctl parameter, in a normal Linux VM, the following command will just work:
root@x:/# sysctl -w net.core.somaxconn=10000    
However, try it in a pod and you get this:
root@x:/# sysctl -w net.core.somaxconn=10000  
sysctl: setting key "net.core.somaxconn": Read-only file system
Now you start realising that this is not your standard Linux VM where you can do whatever you want, this is Kubernetes' turf and you have to play by its rules.

Docker image baking

At this point you're probably wondering why am I not just baking the sysctl parameters into the docker image. The docker image could be as simple as this:
FROM alpine
#Increase the number of connections
RUN echo "net.core.somaxconn=10000" >> /etc/sysctl.conf
Well, the bad news is that it doesn't work. As soon as you deploy the app and connect to the pod, the kernel parameter is still 128. Variations on the docker image baking theme can be attempted, however I could not successfully do it and I have a strong feeling that it's not doable this way.

Kubernetes sysctl

There is documentation around sysctl in Kubernetes here. So Kubernetes acknowledges the fact that kernel tuning is required sometimes and provides explicit support for that. Then it should be as easy as following the documentation, right? Not quite, the documentation is a bit vague and it didn't quite work for me. I'm not creating pods directly as described in the documentation, I am using deployments.

After a bit of research, I did find a way to do it, via init containers. Init containers are specialised Containers that run before app Containers and can contain utilities or setup scripts not present in an app image. Let's see an example:

In order for this to work you will need to create a custom ubuntu image that never terminates. It's a bit of a hack, I know, but the point is to keep the pod running so that we can connect and inspect that the sysctl changes were successfully applied. This is the Dockerfile:

In order to test this we have to build the image first:
docker build -t ubuntu-test-image .
I am using minikube locally, for testing, so I'm creating the deployment like this:
kubectl apply -f ubuntu-deployment.yml
In a Kubernetes cluster, the config has to be provided with the command.
Once the deployment was created let's log into the pod and inspect the changes. First we need to find the pod id:
bogdan@x:/$ kubectl get pods | grep ubuntu-test
ubuntu-test-1420464772-v0vr1     1/1       Running            0          11m
Once we have the pod id, we can log into the pod like this:
 kubectl exec -it ubuntu-test-1420464772-v0vr1   -- /bin/bash
Then check the sysctl parameters:
:/#  sysctl -a | grep "net.core.somaxconn" 
net.core.somaxconn = 10000 

:/# sysctl -a | grep "net.ipv4.ip_local_port_range" 
net.ipv4.ip_local_port_range = 1024     65535
 As you can see the changes were applied. 

Sunday, 24 July 2016

Testing Apache Camel routes

Running Apache Camel on top of Spring is quite popular nowadays and there is a multitude of resources on how to create the routes. However, sooner or later you will want to test the routes and this is where people usually hit a brick wall, because the documentation is a bit confusing. If you're still trying to find a good way to test your routes, then read on.

A simple route

We are using 3 queues: seda:groceries, seda:fruits and seda:vegetables. In case you're wondering what SEDA is, it's just an in-memory queue provided by Camel and it can be replaced by anything else like JMS or RabbitMQ, the principle stays the same.
The type of objects pushed into the groceries queue is checked via Camel's choice() construct and the fruits are redirected into the fruits queue, while the vegetables are redirected into the vegetables queue. The fruits queue is processed by FruitProcessor and the vegetable queue is processed by VegetableProcessor.

The test setup

In order to test the routes, we need to start Camel, which is not possible using the production configuration, considering the fact that Camel will connect to various third party services. Also, we need to make sure that we don't run the real processors, because we only want to test the routes.

This is the test setup, which is actually the complicated part of all this:

A few essential points for the test setup:
  1. We are using CamelSpringJUnit4ClassRunner to run the tests
  2. The test has its own configuration, defined in the ContextConfig inner class, which extends SingleRouteCamelConfiguration
  3. The route() method in ContextConfig() defines the route builder class that we are testing
  4. All the processors are defined as beans and they are mocked by using Camel's MockEndpoint. That allows us to replace the actual processors with mocks and make assertions on how the messages are travelling through the routes.
  5. The mock processors are autowired into the test class so that assertions can be made against them

The tests

After the setup is complete, the actual tests are quite straightforward:
  1. When I push a Fruit object into the groceries queue, I would like the FruitProcessor to process the message
  2. When I push a Vegetable object into the groceries queue, I would like the VegetableProcessor to process the message
This is the actual code:
There are many ways to go about testing the routes, but the setup and test strategy remain the same. I hope you found this useful, if you did then please share. The maven project is available here.

Share on Twitter | Share on Facebook | Share on Google+

Thursday, 12 March 2015

Reduce your git repository size in minutes

It is a fact that your git repository accumulates a lot of history. Even though git was not built for binary files, people do store them in repositories and that contributes to the growth. At a certain point you might be removing binary files and looking back at the history of an image is not something that you do every day. So why not remove all the massive blobs from history? I know, it sounds like you need to rewrite the history and that is dangerous isn't it? Not quite, with a nice tool called bfg.

Right, let's start:

1. Download bfg or install it via brew, yum, etc.

2. Create a bare clone of your git repository:
git clone --mirror git://

3. Create a backup of the repository(just in case)
cp -r big-repo.git big-repo.git_bak

4. Run this:
bfg -b 100K big-repo.git

This will remove all files over 100K, but don't worry, HEAD is protected. There are many other options(including protecting other branches), have a look at their documentation or just run bfg with no arguments to see the options.

5.  Run git gc to actually remove the files

cd big-repo.git
git reflog expire --expire=now --all && git gc --prune=now --aggressive

6. [Optional] Create a new repo where you push the changes. I like to push the changes into a new repo to be 100% sure that the repository is in a good state. Before pushing, change the url for remote "origin" inside big-repo.git/config.

7. Push the changes:
git push

8. Done. Enjoy your lean repository!

Monday, 30 June 2014

Enterprise integration - PHP and Java over Thrift

Sometimes integration between different platforms is a must. In an ideal world you would roll the same platform across all your infrastructure, but in reality that is not always possible. The first thing that springs in mind when you have different platforms is web services such as REST, SOAP or XML-RPC(if you lived in a cave for the past 10 years - no offense to those who still run it for "legacy" reasons). It's natural to think of these solutions, since most people noways will publish their APIs over HTTP. Communicating over HTTP is slow and it's perfectly fine for third-party integrations in most cases, because there are no great expectations performance wise. But what if the integration is done internally and two pieces of your infrastructure, that run on top of different platforms, need to be integrated? Is it acceptable that for every HTTP request to your application you make another HTTP request? If it is, then please stop reading this, I am wasting your time. If it's not then read on.
There is a better way for integration and that is what giants such as Google or Facebook are doing. Google has Protocol Buffers and Facebook has Apache Thrift. Let's have a quick look at both.

What Are Protocol Buffers?

Protocol buffers are Google's language-neutral, platform-neutral, extensible mechanism for serializing structured data – think XML, but smaller, faster, and simpler. You define how you want your data to be structured once, then you can use special generated source code to easily write and read your structured data to and from a variety of data streams and using a variety of languages – Java, C++, or Python.

What is Apache Thrift?

The Apache Thrift software framework, for scalable cross-language services development, combines a software stack with a code generation engine to build services that work efficiently and seamlessly between C++, Java, Python, PHP, Ruby, Erlang, Perl, Haskell, C#, Cocoa, JavaScript, Node.js, Smalltalk, OCaml and Delphi and other languages.

Java+PHP, Protocol Buffers or Thrift?

The answer is obvious, PHP is only supported by Thrift. If you look at the list of languages supported by Thift, it will be the natural choice for many. Facebook have developed it internally, it works great for them at a massive scale, which means it should be brilliant for you also.
In order to be able to use Thrift, you must download it and install it. Please follow the instructions on the official site. Please note that you don't need Thrift installed to run my example, just clone the repository from Github: git clone

Java Server, PHP client

Let's say that we want to consume the services of a Java application from PHP. The reverse is also possible and quite easy to accomplish, as you will see. The Java application is a doughnut vending machine and the client has the possibility of having a look at what's available and ordering.
The most essential part in understanding Thrift is understanding it's Interface Definition Language(IDL). Based on the IDL file(s), Thrift will compile your server and you client in any of the supported languages. Let's have a look at our IDL file:

The above defines the KrunchyKreme service and two methods: getMenu and order. getMenu takes no parameters and will return a list of Doughnut(the definition of Doughnut is begins with "struct Doughnut"). The "order" method allows the client to place an order for a specific doughnutId and a quantity. Please note that the data types are platform independent and they will get translated into the appropriate types depending on the compilation language. You can read about Thrift IDL files here later if you wish.

Now the fun bit, let's compile our Java server and PHP client:

The Java files will be generated into the gen-java folder and the PHP files will be generated into the gen-php folder. In my sample project I have conveniently created a generate-java and generate-php script that also copies the files into the right folder structure for the Java and PHP projects.
You can generate your clients and servers in any of the supported languages, which is pretty good if you ask me. 


The Java server

My sample project contains the java server. In uses maven and you can build it by running the build script bin/ that just executes "mvn package". This is the server:

All this does is starting a single threaded server that handles requests by using the KrunchyKremeHandler class:

If you are wondering where KrunchyKreme.Iface came from, the answer is easy: it is generated by Thrift. So you will get an interface for your server, all you need to do now is write the actual code.  Easy, right?

Once the server is built, you can start it by running ./bin/ in the server's folder. It will start listening on port 9090 and will output debug information when clients connect and call it's services.

The PHP client

You can find the sample client here. The project is based on composer, so you will have to install the dependencies by running php componser.phar install.

The getClient() method initializes the client. Once that is done, getMenu() is called and the three successive orders are placed by calling order(..). Please note the KrunchyKremeClient class, which was generated by Thrift. This gives us all the methods available to call on the server, all we have to do is use it.
You can run the client with php client.php in the client folder. This is the output:

As expected, we display the menu and then output the results of the orders. Please note the response time: 3ms to connect, 3ms to get the menu and 1ms for each order. I am running this on a basic DigitalOcean machine with 1GB or RAM, 1 processor and SSD.  Fair enough, this is a simple example, but try this over HTTP. I don't have numbers for this, but I would expect it to be about ten times worse for Restful web service calls. Not to talk about the resources used, a web server that serves web services will be much more expensive than a simple standalone server. This is why Google and Facebook don't do much Restful internally but Protocol Buffers and Thrift. 

In conclusion, is the complexity worth it? It really depends on your needs, but I believe we will see Thrift more and more.

Friday, 31 May 2013

Updating InfusionSoft Custom Fields

This is a quick post, and I'm writing it mainly because I wasted a lot of time trying to programatically update custom fields in Infusionsoft. So here goes, if you need to update a custom field you need to know the id of the field and the new value:

 updateCustomField($fieldId, $fieldValues)  

You can find the id of the field by going to Admin -> Settings -> Set up custom fields for -> Go and when you hover the field you are interested into you will see the id.
Getting the value is more tricky. What works on any type of field is setting a value manually and then querying DataFormField for the value like this:
 $returnFields = array('DataType', 'Id', 'FormId', 'GroupId', 'Name', 'Label', 'DefaultValue', 'Values', 'ListRows');   
 $query = array('Id' => custom_field_id);   
 $res = $sdk->dsQuery("DataFormField", 10, 0, $query, $returnFields);   
In the result above, have a look at the "Values" field, this is how it should look like.
So to conclude, if you need to update a listbox custom field, this will do it: 

 $values = array(  
     'Values' => "\naaa\nbbb\nddd"  
 $result = $sdk->updateCustomField(custom_field_id, $values);  

Monday, 19 November 2012

Couchbase smart clients

While doing some research on replacing Memcached with Couchbase for the company I work for, I came across the term "Couchbase smart client". The problem is I couldn't find a decent explanation(in my opinion) for what exactly it does. As you will find out next, the smart client is maybe the most important aspect when it comes to connecting to a Couchbase cluster.
Let's have a look first at what the Couchbase documentation says about smart clients:
When using a smart client, the client library provides an interface to the cluster, and performs server selection directly via the vBucket mechanism. The clients communicate with the cluster using a custom Couchbase protocol which enables the sharing of the vBucket map, and the selection within the client of the required vBucket when obtaining and storing information

I couldn't figure out how that is affecting me as a client of the cluster, so I decided to have a look at the low levels calls that a simple client is doing in order to connect to the cluster and write a key. Below is the source code(PHP):

$servers = array('', '');
foreach ($servers as $server) {
        $couch = @(new Couchbase( $server )); // uses the default bucket
        if($couch->getResultCode() === Couchbase::SUCCESS) {

$couch->set('kk', 1);

echo "Done";

Let's run strace  for this script:
strace php ./bin/Temp/couchbase_connect.php   2> /tmp/couch_connect

The output is rather verbose, but we are only interested in the following bits:

  • #1 connect(6, {sa_family=AF_INET, sin_port=htons(8091), sin_addr=inet_addr("")}, 16) = 0
  • #2 sendto(6, "GET /pools/default/bucketsStream"..., 56, 0, NULL, 0) = 56
  • #3 recvfrom(6, "HTTP/1.1 200 OK\r\nTransfer-Encodi"..., 2048, 0, NULL, NULL) = 225
  • #4 connect(7, {sa_family=AF_INET, sin_port=htons(11210), sin_addr=inet_addr("")}, 16) = -1 EINPROGRESS (Operation now in progress)

The above is a massive simplification of what is happening, but it's enough to give us a clue of what the client is doing:

  • at line 1 the client is connecting to the first server in the pool on port 8091
  • at line 2 the client is requesting information about the servers in the cluster
  • at line 3 the client starts receiving that information
  • at line 4 the client is initiating a connection to the server in the pool that is responsible for the key that we want to write, port 11210

To make this short and sweet, the client is connecting to one of the servers in the pool and by receiving information about the cluster, it is able to decide which of the servers is handling the keys it needs. So what makes the client "smart" is the fact that it has knowledge about the cluster and it makes decisions based on the current state of the cluster.

You can start seeing the advantages of such a smart client when you think about not so smart clients like memcached, where you would have to implement all the logic about the state of the cluster on the client side, which is no trivial task by most people's standards.

Saturday, 19 June 2010

PHP remote debugging with Xdebug and Eclipse PDT

Debugging is an invaluable part of software development. I find it very useful in a variety of situations, for instance when I want to understand how a routine works or I need to get rid of a bug that is not exactly easy to fix just by reading the code.

There are several ways to perform debugging in PHP:
  • The most straightforward technique is to use print_r() and var_dump(). This will alter the output, it's quick but very dirty. If you're using this there's nothing to be ashemed of, everyone is doing it.
  • Logging into files/database tables at specific points in the code. This is cleaner than the previous method, but it requires additional effort and usually polutes the code with logging routines. Also this is not exactly debugging, it's logging and analisyng the logs.
  • Using proper debugging tools like Xdebug or the Zend Debugger, integrated into your PHP IDE. This is the clean way to do it, it provides a much better insight into the source code, as you can run it interactively, step by step.
My main goal in this post is to show you how to set your debugging environment with Eclipse PDT and Xdebug. If you're not already using it, get your Eclipse PDT from and install it. Next you will have to get and install xdebug on the machine where PHP runs(it can be the same machine or some remote machine). You should be able to get it through PHP PECL with the following command:

pecl install xdebug

If the above does not work, check the Xdebug installation instructions at .

Once the xdebug extension was installed, you will have to add the extension to php.ini. Add the following lines to php.ini:


On Windows+PHP 5.2.14 I had to replace zend_extension with zend_extension_ts:

Be extra careful with xdebug.remote_host, this is the host where you develop and run your Eclipse, and PHP will try and connect to Eclipse when debugging is enabled. Also make sure that the zend_extension part was not added automatically by the installation, if it was don't add it again.
If there is any mention of the Zend debugger in you php.ini file, you will have to comment that. Restart Apache or whatever web server you're using and make sure the Xdebug installation was correct by running a simple PHP script that contains phpinfo() and searching for "xdebug".

Now the tricky part, Eclipse has to be configured to accept debugging sessions from XDebug. Follow the steps below:
  1. Open your project in Eclipse PDT
  2. In the main menu select Project->Properties
  3. On the left side of the window select "PHP Debug" and then click on "Configure Workspace Settings"
  4. On the "PHP Debugger" dropdown select Xdebug and click "Apply"
  5. Click "Configure" to the right of Xdebug in the same window.
  6. Select Xdebug and click "Configure".
  7. On the "Accept remote session(JIT)" select "any" and click "OK". This is extremely important and this is where most people get stuck.
That's it, Eclipse is now configured, now all we need is to be able to be in control of our debugging sessions. For this we will need to install a Firefox extension called "easy Xdebug"(yes Firefox, you're not developing PHP in IE are you?).

The extension can be installed from . If the link does not work just google "firefox xdebug". Install the extension and restart Firefox. After that you will notice a little green bug on the bottom-right of Firefox and if you hover it it says "Start xdebug session".

As a side note, you might have used Zend IDEs where the debug process starts from the IDE. In Eclipse PDT the process is reversed: you start from the page that you want to debug and PHP will connect to Eclipse in order to establish a debug session. That is why we have installed the firefox extension, because the debug starts from the browser.

Now open the page that you want to debug, on the server where you have just configured PHP with the XDebug extension of course. Click on the green bug I just mentioned to enable debugging and then reload the page. After this you will have to go to Eclipse and see that a new window has just popped up, asking you to "Select the local resource that matches the following server path". In a simple setup you will have just a single option, select the PHP file in that window and click "OK". Eclipse will ask you if you want to change to "PHP Debug perspective" and obviously you have to say "Yes". Optionally you can also check "Remember my decision". After this you should be in the debugging perspective, with Eclipse stopped on the first line of your code, meaning that you can now step through your code.

As a simple guideline you can use the following keys:
  • F5(Step Into) - steps into everything including function or method calls
  • F6(Step Over) - walks through but does not step into function or method calls
  • F8(Resume) - runs until the first breakpoint or end of the program
Breakpoints can be placed by double-clicking on the right of the line where you need the breakpoint. Try and play with the above keys to get a better idea of how they work.

That's it, I'm sure you'll realise that you can't live without debugging once you start using it.