The first thing which impressed myself was the amazing location. It happened at the Lucerna Cinema in Prague; for whom doesn’t know what I’m talking about just have a look at the pictures.

Overall it was very well organized and they improved things even while the event happened. For instance on the first day there wasn’t enough space to have proper lunch but this was different on the second day by opening an additional room. Well done! Speaking of food: as at most of the developer / computer science conferences there was plenty and good food, coffee and drinks. Yummy.

But surly I wasn’t attending because of food and drinks. Talkwise there was a big variety in terms of topics but sadly there wasn’t a big diversity in terms of gender. There were apparently many more men talking than women. Surprisingly to me especially because I had the feeling that the proportion of women attended the conference was pretty high compared what I’ve seen at developer conferences so far. So I’m convinced to see more women on stage next year.

We have more than 500 people from 43 countries at @mlprague. It's gonna be great! pic.twitter.com/qvwnqEtumJ

— Jiří Materna (@JiriMaterna) April 22, 2017

Since there where too many talks as I can write up in a reasonable time I will only pick some I was particular excited. Please bear with me in case I skipped yours.

Vertical AI: Solving full-stack industry problems that require subject matter expertise, unique data, and a product that uses AI to deliver its core value proposition

By Bradford Cross, DCVC (USA). This talk inspired me a lot. He mentioned that engineers tend to apply the engineering knowledge first to their very own craft. This seems to me somehow true and at the same time it’s an opportunity to change it. Bringing engineering / science mindsets to a variety of industries will help to add another dimension to see things from a totally different angle.

The only problem I see in this path is that in my feeling only tech companies appreciate the value of engineers and scientists enough and therefore a lot of companies don’t have engineers at c-level. My personal assumption is that companies with people having an engineering/science background in the c-level will overcome the competition in the long run because they making use of more advanced technology and science. But this are just my personal thoughts and where not part of the talk.

Data Science at The New York Times

By Chris Wiggins, New York Times (USA) – my personal favorite. He gave good general advice and was inspiring at the same time. For example he mentioned that they integrated their Machine Learning tool as slack bot. With it you get the power of ML to marketies and journalists without friction. Here a selection of tweets which reflect very well what I mean:

Data science at The New York Times or "Every publisher is now a startup". #mlprague pic.twitter.com/fNps2D4EDH

— Kateřina Veselovská (@kveselovska) April 22, 2017

@chrishwiggins #mlprague descriptive, predictive, prescriptive -> use the language people get :) pic.twitter.com/eKlB2yPxmF

— Andrej Danko (@DankoAndrej) April 22, 2017

Microsoft Cognitive Toolkit: fast and furious

By Willi Richert, Microsoft (DE). MCT seems to be a good tool in anyones belt so give it a try and figure out for yourself if it’s helpful for your particular need or not. The talk itself gave a basic overview what I personally liked.

Willi Richert @derpapst presenting the @Microsoft Cognitive Toolkit #deeplearning framework at #MLPrague #machinelearning pic.twitter.com/TumTUzwHfQ

— Tom Lidy (@LidyTom) April 22, 2017

Predicting short term profit on US stocks by multiple ML methods

By Michal Illich, Wikidi (CZ). This talk had an unexpected twist for a machine learning conference. But I think it was even a good thing to show that there is still more than machine learning what solves problems. He made a good point and hearing the story about the path they went was very worth to listen.

Funny slide especially at a machine learning conference. #MLPrague pic.twitter.com/5ql1rg1VUh

— Simon Dittlmann (@SimonDittlmann) April 22, 2017

Serving a billion personalized news feeds

By Lars Backstrom, Facebook (USA). Since Facebook has both a big variety of data per person and a lot in total I was looking forward to get some insights what they are doing. He explained briefly the journey they made from decision trees to neuronal networks. I’d liked the talk very much.

Lars Backstrom from @Facebook explaining how they rank new stories for users on facebook using #machinelearning at #MLPrague #AI pic.twitter.com/D8K9EKiVPn

— Tom Lidy (@LidyTom) April 22, 2017

And one funny exploration…

Serving a billion personalized news feeds #mlprague
Facebook looking at your data can predict chance your relationship end in next 60 days. pic.twitter.com/tEvUhhpwiU

— Kamil Krzyk (@F1sherKK) April 22, 2017

Neuron soundware

By Martin Krivanek (Neuron soundware). “We use IoT devices to listen for the sound of industrial machines and use deep learning and power of neural networks to recognize their failures.” Nothing more to add besides that this is really a cool idea.

Using sounds to determine problems of machines like for example elevators. Great idea. #MLPrague pic.twitter.com/xcZREOngBA

— Simon Dittlmann (@SimonDittlmann) April 22, 2017

The road towards chat automation starts with humans

By Janos Szabo (Chatler). “How human chat agents can help chatbots to get smarter, and how ML can help human chat agents to be more productive.” Since chat bots aren’t as good as humans when it comes to customer service yet it’s definitely a good idea to combine the best of both worlds. Not sure if Chatler is the best product doing this but certainty the idea it reasonable.

TensorFlow & Deep Learning

By Yufeng Guo, Google (USA). Tensorflow is so famous lately of course there has to be a slot for it. Good overview, thank you Guo!

Creating adaptive worlds where people experience imagination

By Maria Vircikova, Matsuko (SK).

Interesting talk by @marivirci from #Matsuko about virtual-hologram friend, want one right now :) #MLPrague https://t.co/I0LX1BAohP …

— Lubomír Šerý (@gugatr0n1c) April 23, 2017

Towards Good AI

By Roman Yampolskiy, University of Louisville (USA). Clearly an important topic to discuss the ethical aspects of AI and especially of Artificial General Intelligence. I remember he said something like we will have basically two options: a) we get uploaded somehow to such a system or b) we are becoming Cyborgs. In case of (a) we are loosing most of the things a human is today (eating, sex, etc) in case of (b) the system will get rid of us when it’s getting too smart. Who attaches a 486 PC to an iPhone? Nobody does. Why should a super intelligence attach a human in the long run?

Certainly a lot of worth points to be discussed.

"Children are untrained neural networks deployed on real data." @romanyam about #AI and #security and near future #MLPrague pic.twitter.com/hXBfil1V0q

— Stanislav Rejthar (@s_rejthar) April 23, 2017

DeepStack: Expert-Level Artificial Intelligence in No-Limit Poker

By Martin Schmid, IBM (CZ). Too bad for online Poker: DeepStack beat single Poker games now after years of research. Congratulations for that and thank you for sharing with us in this talk!

DeepStack team... pretty interesting piece of research. #mlprague pic.twitter.com/tgtXAjejL4

— Pavel Suchmann (@bver_en) April 23, 2017

Multi-Instance Learning in Security

By Tomas Pevny, Cisco (CZ). How to apply AI in network security could be seen in this talk. And analysing network traffic is in particular harder than normal AI problems because you don’t have just one vector as training set, you have multiple different ones. I have to emphasis that his Q/A session was in particular informative and a little bit funny at the same time. At least I was awake afterwards again.

Please @MLPrague bring #TomasPevny from @cisco next year again. Great speech - fun and very informative #MLPrague #multiinstancelearning

— Lubomír Šerý (@gugatr0n1c) April 23, 2017

Summary

So far so good. You can find the official schedule with all abstracts at http://www.mlprague.com/#schedule. I hope you continue the great conference and see you next year.

"We haven't actually studied how it works. It just does." 😎 #MLPrague pic.twitter.com/F1C1V0oZC3

— Ondřej Veselý (@xorwen) April 22, 2017

In the end some more ML/AI related links:

• http://www.mlmu.cz/
• http://www.ceai.io/
• https://www.microsoft.com/en-us/research/product/cognitive-toolkit/
• https://www.h2o.ai/
• http://ai-valley.com/
• https://www.knime.org/
• http://course.fast.ai/
• http://cyber-valley.de/
• https://www.deepstack.ai/

Overview

We are going to build a modern delivery pipeline starting with building and unit/integration testing the software. Afterwards we will deploy the current feature branch to a new one-time staging environment. Gitlab calls this a review app. And as soon as it’s ready it will be integrated and deployed to a staging system for final integration testing. As soon as you’d like to release just tag the commit in git and the exact docker image will be tagged the same way. Finally the release is going to production and monitoring it is now the number one priority. Luckily we have Stackdriver Logging on google cloud for all log file analysis. For server/kubernetes monitoring you can use for example Datadog. How the workflow looks like from a branching perspective you can see here:

And these are the tools we will use within this flow:

• Gitlab: Git repository
• Gitlab-CI: continuous integration and deployments
• Google Container Registry: docker images
• Google Container Engine (GKE/Kubernetes): our application cluster (one or multiple clusters)
• Google Cloud: Logging
• Datadog: Performance Monitoring

Enough talking. Let’s get it done. Just within the next 60 minutes.

Setup, step by step

This step by step instructions will help you to get Google Cloud Platform, Google Container Engine (GKE/Kubernetes) up and running and wire everything with a gitlab installation – even gitlab.com.

Basis: Google Cloud Platform

1. The first thing you need to do is to create your own Google Cloud Platform Account. With it you are able to use all services. Be aware you can sign-up for a trial of 60 days and with enough credits to play around. It’s free, no strings attached.
2. After you’re logged into your fresh Google Cloud account you should start the Cloud Shell. Check the quick-start documentation if you need more assistance.
3. Now we are ready to setup our demo application. To do so clone the following git repository git clone https://github.com/Pindar/gcloud-k8s-express-app.git && cd gcloud-k8s-express-app/
4. Setup GCloud project ./k8s/gcp/setup_gcloud_project.sh [test-gitlabci-k8s-XXX]. This creates a new google cloud project and activates billing for you as long as you are using a German account otherwise you need to change ACCOUNT_ID=${2:- ` gcloud alpha billing accounts list | grep "Mein Rechnungskonto" | awk '{ print $1 }' ` } at https://github.com/Pindar/gcloud-k8s-express-app/blob/master/k8s/gcp/setup_gcloud_project.sh#L11
5. Since you have a working Google Cloud Project the next step is to setup all required google cloud resources. In particular we need a Kubernetes cluster where we are going to deploy our application later onto. Furthermore we need a service account with restricted access to our account which is going to be used for deployments later on. To make both things as simple as possible you can just run the setup script with the project name as parameter: ./k8s/gcp/setup_gcloud_resources.sh [test-gitlabci-k8s-XXX]

Continuous Integration: Gitlab-CI

1. Register at Gitlab
2. Create your own repository and clone example repository https://github.com/Pindar/gcloud-k8s-express-app.git
3. Now we need to prepare Gitlab-CI. The first step is to prepare all required variables. A list is shown in the table below. To make your life easier you can run create_gitlab_variables.sh [test-gitlabci-k8s-XXX] where the first argument is again the google cloud project where it belongs to. For updating later on you can just call update_gitlab_variables.sh [test-gitlabci-k8s-XXX]. The scripts need all variables defined within a .gitlab-env file. So don’t forget to create the file before running the script!

1. If you don’t trust the shared gitlab-ci-runner on gitlab.com check out this related how-to to have your own runners on your GKE/Kubernetes cluster.
2. It’s done. Now you can run a build. Make a small change at any file, commit, push and see how the pipeline kicks-in.

DNS / Connecting Cluster

1. As soon as master was build once there will be a ingress load balancer which is going to have the public IP address. Therefore your last setup-step is to update your DNS settings to the ingress IP you get by calling kubectl --namespace=production get ing
2. Now you can also proxy to your Cluster from your Cloud Console. First run kubectl proxy --port 8080 and second start web preview, see

In case you run it locally on your pc or mac

1. configure kubectl gcloud auth application-default login && gcloud --quiet container clusters get-credentials example-cluster --zone europe-west1-b && export KUBERNETES_CONFIG=~/.kube/config
2. [OPTIONAL] activate context kubectl config get-contexts, kubectl config use-context [CONTEXT_NAME]

Use it

Now it’s time to create an issue at gitlab. Create a branch for it starting with the issue number and do some changes. When you’re done push it to gitlab and create a merge request. You will see that Gitlab-CI will shortly afterwards start building/testing and deploying the changes as review app to your Kubernetes cluster. And as soon as you merge these changes to master and remove the feature-branch the review-app is going to tear down automatically and all the joy I was talking in the beginning is in place. Wasn’t that easy?

Clean up

To clean everything up again afterwards just call the following scripts which are doing basically the inverts of what we have done during the setup.

kubectl config

1. Remove context; list them kubectl config get-contexts, remove kubectl config delete-context [CONTEXT_NAME]
2. Remove cluster; list them kubectl config get-clusters, remove kubectl config delete-cluster [CLUSTER_NAME]

project/resources

1. ./k8s/teardown.sh
2. ./k8s/gcp/tear_down_gcloud_resources.sh [test-gitlabci-k8s-XXX]
3. ./k8s/gcp/tear_down_gcloud_project.sh [test-gitlabci-k8s-XXX]

Summary

I hope you’ve enjoyed this short blog post how to setup quickly a continuous integration/delivery pipeline with Gitlab and Kubernetes on Google Cloud Platform. If you find any bugs or have any question just write me in the comments or on twitter.

Links

• Slides https://www.slideshare.net/Pindar/idea-to-production-with-gitlab-and-kubernetes
• Code https://github.com/Pindar/gcloud-k8s-express-app

Real contextual information is possible without being disturbed by push notifications. For instance you are waiting in public office area. Now you can pick up a digital ticket just as you enter or you get finally the online self care service shown. An other good show case might be a train station which shows you the delays etc. Means there are plenty of opportunities!

The good news are you can set-up a basic physical web experiment just within minutes if you have a Raspberry PI 3 with Jessie:

1. Enable Bluetooth sudo hciconfig hci0 up
2. Advertise and not-connectable sudo hciconfig hci0 leadv 3
3. Generate Eddystone URL and call it http://yencarnacion.github.io/eddystone-url-calculator/ (only HTTPS target URLs are working: http -> https works, https -> http doesn’t)

How to enable Physical Web beacons on iOS.

And even more information and ideas you will find in this video:

In times of microservices and single responsibilities it has to be a better way to deal with it and in fact after doing some research (also talking with pkircher on the coreos IRC channel) I came up with the following:

1. Registrator (as docker container) runs on each server once and automatically register/deregisters services for Docker containers based on published ports and metadata from the container environment.
2. SkyDNS (as docker container) runs on each server once and provides DNS to discover available services.
3. Logspout (as docker container) runs also on each server once and ships docker logs to a centralized syslog server.

When a service comes online the following happens:

The big advantage with this setup is that both the service discovery and the log shipping (through logspout) are completely decoupled from the application and still everything is dockerized - no further requirements on the host.

The simplified elasticsearch systemd-unit file can look like this:

[Unit]
Description=ElasticSearch service
After=docker.service

[Service]
TimeoutSec=0
EnvironmentFile=/etc/environment

ExecStartPre=/usr/bin/mkdir -p /vol/data/elasticsearch
ExecStartPre=/usr/bin/docker pull dockerfile/elasticsearch

ExecStart=/bin/bash -c '\
  exec docker run \
  --name %p-%i \
  -h `hostname` \
  --publish 9200:9200 \
  --publish 9300:9300 \
  --volume /vol/data/elasticsearch:/usr/share/elasticsearch/data \
  -e SERVICE_ID=%p%i \
  -e ES_HEAP_SIZE=512M \
  --dns ${COREOS_PRIVATE_IPV4} \
  --dns-search=example.local \
  elasticsearch:1.5.2 \
  elasticsearch \
  --node.name=%p-%i \
  --cluster.name=logstash \
  --network.publish_host=${COREOS_PRIVATE_IPV4} \
  --discovery.zen.ping.multicast.enabled=false \
  --discovery.zen.ping.unicast.hosts=elasticsearch-9200.staging.example.local'

ExecStop=/usr/bin/docker stop %p-%i
ExecStop=/usr/bin/docker rm %p-%i

[X-Fleet]
Conflicts=%p@*.service

It’s just a simple reference to the unicast host of the “load balanced” skydns image route. Because one of the nice out of the box features of skydns is that it provides automatically round robin balancing of multiple registered docker-container from the same docker-image.

You can find the updated code about the rsyslog/elasticsearch/kibana setup on github.

And because I like it this is written with StackEdit.

When you containerize your applications you will find out that you can do so much more with container and in particular with docker than just running your applications. You can use docker in all stages for different purposes:

Development

• prepare a docker image with all required development tools for a certain project. With it it’s easy to ramp up new team mates and keep everyone in sync.
• of course dockerize your application and use the dockerized version to be close to production even in the first stage. The downside is that this doesn’t work well on MacOS or Windows yet. On MacOS you can have docker running in a VM but the mounting of the host files are causing performance issues even with NFS.
• run your databases within a container. This makes it easy to start from predefined states when you have some import scripts prepared. One use case is to test data migrations easily.

Continuous Integration

• use the tooling container you already have for development also on your continuous integration server. Best case you don’t need to install any other tool than docker on the host which makes all ci-jobs totally independent of each other. This is very handy if you have only one Jenkins Server but it has to run jobs for a lot of different teams.
• use docker-compose for continuous integration testing. With it you can for example start your application together with a dockerized database and a third container runs the end to end tests against both. As long as you have enough memory and CPU you can bring up even more complex environments. Note that you can run these tests even on open pull requests.

Staging

• use Kubernetes, CoreOS or docker-compose to start new environments quickly – for example one per git branch.
• use centralized application logging systems like kibana or loggly to find bugs early. Dockerized applications can easily log to stdout and these logs can be shipped by logspout to a centralized syslog server or since docker 1.6 you can directly ship to syslog. To make the analysis simpler you should have a look at the @CEE logging project.
• use performance logging tools for example datadog. Consider that you need to think in layers which means you need to monitor the underlying infrastructure independently of your application layer. With a proper service discovery and an overlay network you can centralize the gathering process of all application metrics.

Production

• just use the same things as in your staging environment.

Summary

These are just a few ideas how you can use docker in your continuous delivery pipeline. A general recommendation is to have your own self-hosted private docker repository or at least a docker registry cache running to avoid problems when your third party docker-registry is unavailable. These caches are speeding up the deployment times a little bit as well and reducing the bandwidth consumption which reduces your costs a little bit too.

Photo is under creative commons

Centralized logging is a key feature of a modern platform architecture. Obviously CoreOS brings some CLI tools to get logs but usually you are not in the lucky position to have only a coreos cluster running. Instead you probably have to manage a lot of different applications on different systems.

My goal was to setup a centralized log system quickly and without writing too many custom scripts. To achieve these two goals I tried Opsworks, Saltstack and in the end coreos. And I have to admit that the simplest approach for me was to use a coreos cluster. In this blog post I like to guide you through the setup step by step and also explain the log shipping system briefly.

You can find all the code on github in my coreos-demo project.

Log shipping system overview

I thought the best approach to handle a tremendously amount of log messages is to use syslog as shipping mechanism and elasticsearch together with kibana for visualization.

Each application will ship the logs to a central rsyslog server by using the syslog protocol. The good thing is that almost every application logger system supports syslog (log4j, monolog etc.). In case you have application container (container with only one application running) you can also use a separate docker container that ships the logs. The important thing is to tag each log entry for simpler log parsing in the next step. Now the central rsyslog server can do both parse each log message and forward each log message to elasticsearch. Finally you can visualize and search through the log entries with kibana.

Hands On

Let’s setup the following:

• running rsyslog service
• make rsyslog available through a private route53 dns entry (e.g. syslog.foobar.local)
• running elasticsearch cluster
• running kibana which points to the elasticsearch cluster
• make the kibana service available through a private route53 dns entry (e.g. kibana.foobar.local)

Elasticsearch

Let’s start with the elasticsearch cluster because both rsyslog and kibana depend on them. Luckily you can find an official ES image on dockerhub which means the only thing we need to do is writing the required systemd files. As reference implementation I found the instructions of Matt Wright

I’m almost using Matt’s approach. Only I’m using my own announcement mechanism which does health checking in addition and mounting the data to a different volume on the host. In the best case this volume is a EBS volume and gets attached to the instance through another service at runtime. But for simplicity I skip this part.

[Unit]
Description=ElasticSearch service
After=docker.service

[Service]
TimeoutSec=180
EnvironmentFile=/etc/environment

ExecStartPre=/usr/bin/mkdir -p /vol/data/elasticsearch
ExecStartPre=/usr/bin/docker pull dockerfile/elasticsearch

ExecStart=/bin/bash -c '\
  curl -f ${COREOS_PRIVATE_IPV4}:4001/v2/keys/announce/services/elasticsearch/logs; \
  if [ "$?" = "0" ]; then \
      PEER_PATH=$(etcdctl ls /announce/services/elasticsearch/logs | head -1); \
      UNICAST_HOSTS=$(etcdctl get $PEER_PATH | awk \'/:/ { print $2 }\' | cut -d\'"\' -f 2); \
  else \
      UNICAST_HOSTS=""; \
  fi; \
  /usr/bin/docker run \
    --name %p-%i \
    -h `hostname` \
    --publish 9200:9200 \
    --publish 9300:9300 \
    --volume /vol/data/elasticsearch:/data \
    -e ES_HEAP_SIZE=512M \
    dockerfile/elasticsearch \
    /elasticsearch/bin/elasticsearch \
    --node.name=%p-%i \
    --cluster.name=logstash \
    --network.publish_host=${COREOS_PRIVATE_IPV4} \
    --discovery.zen.ping.multicast.enabled=false \
    --discovery.zen.ping.unicast.hosts=$UNICAST_HOSTS'

ExecStop=/usr/bin/docker stop %p-%i
ExecStop=/usr/bin/docker rm %p-%i

[X-Fleet]
Conflicts=%p@*.service

Save this systemd configuration to elasticsearch@.service and submit it as systemd template to your CoreOS cluster by typing fleetctl submit elasticsearch@.service. At this point in time you can already start your elasticsearch cluster assuming your coreos cluster works and the required fleet metadata is set. Do so with fleetctl start elasticsearch@1.service and check the status with fleetctl status elasticsearch@1.service. To get the logs of our elasticsearch container you can use fleetctl again: fleetctl journal elastichserach@1.service.

Let’s announce the new service.

[Unit]
Description=ElasticSearch announce service
After=elasticsearch@%i.service
BindsTo=elasticsearch@%i.service

[Service]
EnvironmentFile=/etc/environment

ExecStartPre=-/usr/bin/docker kill elasticsearch-announce-%i
ExecStartPre=-/usr/bin/docker rm elasticsearch-announce-%i
ExecStartPre=/usr/bin/docker pull pindar/announcement-health-service

ExecStart=/bin/sh -c '\
/usr/bin/docker run \
--name elasticsearch-announce-%i \
-e SERVICE=elasticsearch \
-e THRESHOLD=5 \
-e TIMEOUT=45 \
-e ENVIRONMENT=logs \
-e NUMBER=%i \
-e HOST_IP=${COREOS_PRIVATE_IPV4} \
-e HEALTH_URL=http://${COREOS_PRIVATE_IPV4}:9200 \
-e ANNOUNCE_VALUE=\'{"IP": "${COREOS_PRIVATE_IPV4}", "PORT": "9200", "TRANSPORT_PORT": "9300" }\' \
pindar/announcement-health-service'

ExecStop=/usr/bin/docker stop elasticsearch-announce-%i

[X-Fleet]
MachineOf=elasticsearch@%i.service

Just use the same fleetctl commands as above to start the announcement service.

ElasticSearch Client node

To have a smart load balancing for your elasticseach cluster you can easily start client nodes. Afterwards you can connect kibana or rsyslog to this nodes instead of using the cluster directly.

[Unit]
Description=ElasticSearch client service
After=docker.service

[Service]
TimeoutSec=180
EnvironmentFile=/etc/environment

ExecStartPre=/usr/bin/mkdir -p /vol/data/elasticsearch
ExecStartPre=/usr/bin/docker pull dockerfile/elasticsearch

ExecStart=/bin/bash -c '\
  curl -f ${COREOS_PRIVATE_IPV4}:4001/v2/keys/announce/services/elasticsearch/logs; \
  if [ "$?" = "0" ]; then \
      PEER_PATH=$(etcdctl ls /announce/services/elasticsearch/logs | head -1); \
      UNICAST_HOSTS=$(etcdctl get $PEER_PATH | awk \'/:/ { print $2 }\' | cut -d\'"\' -f 2); \
  else \
      UNICAST_HOSTS=""; \
  fi; \
  /usr/bin/docker run \
    --name %p-%i \
    -h `hostname` \
    --publish 9200:9200 \
    --publish 9300:9300 \
    --volume /vol/data/elasticsearch:/data \
    dockerfile/elasticsearch \
    /elasticsearch/bin/elasticsearch \
    --node.name=%p-%i \
    --node.data=false \
    --node.master=false \
    --cluster.name=logstash \
    --network.publish_host=${COREOS_PRIVATE_IPV4} \
    --discovery.zen.ping.multicast.enabled=false \
    --discovery.zen.ping.unicast.hosts=$UNICAST_HOSTS'

ExecStop=/usr/bin/docker stop %p-%i
ExecStop=/usr/bin/docker rm %p-%i

[X-Fleet]
Conflicts=%p@*.service

You can find all configuration files also on my github account in the demo project – you need two announcement services for the elasticsearch client nodes: one that announces the node in the es-cluster and one that announces the node as client node for other services.

Kibana

Let’s connect Kibana to our new elasticsearch instance. You can find a dockerized version of kibana 4 on my github repo and an automated build on dockerhub.

To start a container from this image we need again the systemd configuration for our coreos cluster.

[Unit]
Description=Kibana logs front-end
After=elasticsearch-announce@*.service
Requires=elasticsearch-announce@*.service

[Service]
EnvironmentFile=/etc/environment
TimeoutStartSec=0
ExecStartPre=-/usr/bin/docker kill kibana
ExecStartPre=-/usr/bin/docker rm kibana
ExecStartPre=/usr/bin/docker pull pindar/kibana

ExecStart=/usr/bin/bash -c '\
curl -f ${COREOS_PRIVATE_IPV4}:4001/v2/keys/announce/services/elasticsearch-lb/logs/; \
  if [ "$?" = "0" ]; then \
      ELASTICSEARCH_ENDPOINT="http://$(etcdctl get /announce/services/elasticsearch-lb/logs/1 | awk \'/:/ { print $2 }\' | cut -d\'"\' -f 2):9200"; \
  else \
      ELASTICSEARCH_ENDPOINT=""; \
  fi; \
/usr/bin/docker run \
--name kibana \
-e ELASTICSEARCH_ENDPOINT=$ELASTICSEARCH_ENDPOINT \
-p 5601:5601 \
pindar/kibana'

ExecStop=/usr/bin/docker stop kibana

[X-Fleet]
Conflicts=kibana.service

As soon as kibana is running you can try to connect to it on port 5601 and should see the interface.

To make it more convenient to access the UI you can dynamically update your DNS server. In case you are using AWS Route53 even for private entries you can use my fork of an implementation that exactly does it for you. The fork was necessary because I don’t wont to remove the dns entry every time fleet moves the service because it takes some time that the entry gets visible again (TTL of the SOA).

[Unit]
Description=kibana announcement service

After=docker.service
BindsTo=kibana.service

[Service]
ExecStartPre=-/usr/bin/docker kill %p
ExecStartPre=-/usr/bin/docker rm %p
ExecStartPre=/usr/bin/docker pull pindar/go-route53-presence

ExecStart=/usr/bin/bash -c \
"/usr/bin/docker run \
--name %p \
-e AWS_ACCESS_KEY=`etcdctl get /AWS_USER_ROUTE53_KEY` \
-e AWS_SECRET_KEY=`etcdctl get /AWS_USER_ROUTE53_SECRET` \
-e ROUTE53_RECORD_NAME=logs.example.local. \
-e ROUTE53_RECORD_TYPE='A' \
-e ROUTE53_TTL=15 \
-e ROUTE53_ZONE_ID=`etcdctl get /AWS_ROUTE53_ZONE_ID` \
-e ROUTE53_IP_TYPE=private \
pindar/go-route53-presence"

ExecStop=/usr/bin/docker stop %p

[X-Fleet]
MachineOf=kibana.service

Rsyslog

The last missing piece is the rsyslog container which will handle all the shipped logs. Even for this you can find a ready to use docker image on dockerhub

[Unit]
Description=Centralised RSyslog
After=docker.service

[Service]
User=core
TimeoutStartSec=0
EnvironmentFile=/etc/environment

ExecStartPre=-/usr/bin/docker kill central-rsyslog-%i
ExecStartPre=-/usr/bin/docker rm central-rsyslog-%i
ExecStartPre=/usr/bin/docker pull pindar/docker-rsyslog
ExecStartPre=/usr/bin/sudo /usr/bin/mkdir -p /vol/logs

ExecStart=/usr/bin/bash -c '\
ELASTICSEARCH_ENDPOINT="$(etcdctl get /announce/services/elasticsearch-lb/logs/1 | awk \'/:/ { print $2 }\' | cut -d\'"\' -f 2)"; \
/usr/bin/docker run \
--name central-rsyslog-%i \
-p 514:514 \
-p 514:514/udp \
-e ELASTICSEARCH_HOST=$ELASTICSEARCH_ENDPOINT \
-v /vol/logs:/var/log/remote \
pindar/docker-rsyslog'

ExecStop=/usr/bin/docker stop central-rsyslog-%i

[X-Fleet]
Conflicts=central-rsyslog@*.service

To test the round trip lookup the ip where rsyslog is running fleetctl list-units and connect with telnet to write your first log message.

telnet 172.17.8.101 514
Trying 172.17.8.101...
Connected to 172.17.8.101.
Escape character is '^]'.
test foobar : this is my message
^]
telnet> quit
Connection closed.

Now everything is up and running – happy shipping!

Recently the micro-service architecture style is growing a lot but even without such an architectural approach you probably have to run a lot of different software components in your data center to provide a simple application. Just think about what you need for a single application you might develop. Your stack could look like the following:

• DNS entries
• IP routing
• firewall settings
• Virtual Machine / Server
• operating system
• load balancer
• apache/nginx
• tomcat with your application running
• database
• log aggregation service
• performance metrics

This means even a simple application needs some other services to run properly. In a more classic approach you would setup all the required software once and for each deployment you would replace just your developed binary (e.g. your war file). To have a more scalable approach it’s good to start with a short planning. Let’s try to sort each of the listed pieces in one of the three following layers (more information):

1. Visible (changes rarely): DNS entries, virtual machines, operating system, firewall settings, ip routing
2. Volatile (changes often): load balancer, apache, tomcat, database software, log aggregation software, performance metrics software, internal dns
3. Persistent (no changes): database data, log data, performance data

Since we have now a clear picture which parts of the system we’d like to change more often and fast we can think about how to achieve it.

Let’s start with the Visible layer.

Visible Layer

This layer captures all pieces changing from time to time but might affect the entire system. The risk to do a change is obviously high.

When you are a happy AWS customer and you’d like to have your infrastructure written in code then Cloudformation is probably your friend to handle this layer. You can write and test it with a separate stack even in a continuous integration process and update your production infrastructure after all testing is done.

But this layer is not the one with the highest risk. You should be really scarred while doing changes on the persistent layer.

Persistence layer

Almost every single application needs to persistent data in the end. Data is usually the most valuable thing for a service and that’s why it should be treated carefully. You probably do backups, never format the volume, do regular checks etc.

With AWS I’m using EBS volumes for this. It’s perfect because you can move the volume from one instance to another one by attaching and detaching it. In addition you can take regularly snapshots and store it to s3.

To process all the data we need an application which probably will change rapidly and that’s why it’s in the volatile layer.

Volatile layer

This layer is for all running applications regardless of whether it’s developed by yourself or just managed. The reason why I don’t differentiate is that you might need to update your managed application because of security reason fast and without hassle. Why should it be harder to deploy a third party application then something developed by yourself?

To keep things simple I’m containerize everything and try to follow the single responsibility rule which means a docker container should only have one process but multiple docker container might orchestrate one single service (also called pod). With this approach it’s quite simple to update each part separately. Sadly docker has some downsides when orchestrating a pod but since the App Container Specification addresses most of them it will be just a matter of time that either Rocket (which implements APPC) or Docker (which doesn’t implement APPC but it’s developed by so many smart people) will solve them.

Nevertheless a containerized application is much simpler to deploy then a traditional one which makes it also simpler to manage your volatile layer.

In the next blog post I will explain how I created a CoreOS cluster with ElasticSearch, Kibana and Rsyslog while following this theories.

• Deploying pre-provisioned AMIs
• Deploying new instances and polling all the docker container on startup
• Opsworks with Docker-Chef recipes
• Saltstack (only as POC)
• Kubernetes (only as POC)

Kubernetes is really nice but “Kubernetes is in pre-production beta!”. And especially it’s stated that you shouldn’t use it with a high load. Also AWS Container Service is not ready for production yet.

So exciting docker is it’s still a very young technology and I wanted to have a toolbox for it which is very flexible and let you deploy new software within minutes. One hard requirement to me is a easy configuration management. So I finally took a look at CoreOS again. And what I have seen so far is really nice. It focus really on “massive server deployments” and gives you all tools you need to get up and running in hours (including learning time).

I started my CoreOS journey with this article about AWS deployments . And got our entire staging system running within one day. But there are also some minor things I’d like to have before it goes to production:

• blue/green deployments
• rolling deployments
• service announcement for discovery
• for some services a 1:1 mapping between AWS autoscaling and started container

The nice thing is that the side-kick service idea let’s you do all of it with a couple of microservices. For the blue/green deployments I plan to update the port mapping on AWS ELBs. The new container will start on a different but fixed port and as soon as it’s running the ELB will be updated. The rolling one is quite simple with fleet. Just a couple of bash lines. For the Service Announcement/discovery I started also a small micro service and it’s open sourced. Not totally done yet but I’m working on it (pull requests are welcome). And the hardest part is my “auto scaling”. The idea is to store the desired amount of started containers for each service in etcd. This value is checked by a microservice which starts or stops the services. While a deployment happens this is obviously paused. The desired value can then come from different sources. One might be an AWS autoscaling group; or you calculate the load based on some metrics you have in e.g. Datadog. No matter how I will open source this little microservice soon.

1. I don’t want to wait until the backend is implemented if I can not do it by myself
2. I like to work offline even in trains when I have no reliable internet connection
3. I like the speed of the JS development process without restarting huge backend systems

All together ended in the idea to mock every backend call while developing JS frontends. The first time I was using this approach I was working in a big project where the frontend team had to start the implementation without any ready backend. This project was month before AngularJS was ready to use and we were working with BackboneJS and jQuery. But we got it working even with this setup and we loved the new way to work. So I’m using the same idea in my current AngularJS + Yeoman setup.

Overview

The basic idea is that if you run grunt serve you get an offline development environment where all backend calls are mocked. But when you build your release grunt all backend calls are real.

How

I’m bootsrapping a “different” dev-AngularJS app when starting it in development mode then when starting it in production. The difference is that this dev-app uses the production app as module, uses ngMockE2E and $httpBackend to mock all backend calls.

Step by step

To see this steps better in a context a demo project is available on github.

1) Add all required packages to your package.json

{
  "name": "mockedbackendwithangularjs",
  "version": "0.0.0",
  "repository": {
    "type": "git",
    "url": "https://github.com/Pindar/mocked-backend-with-angularjs.git"
  },
  "dependencies": {},
  "devDependencies": {
    ...
    "grunt-processhtml": "^0.3.3"
  },
  "engines": {
    "node": ">=0.10.0"
  },
  "scripts": {
    "test": "grunt test"
  }
}

and bower.json

"devDependencies": {
  "angular-mocks": "~1.2.0"
},

then call npm install && bower install

2) Configure grunt-processhtml. Add the following to your Gruntfile

processhtml: {
  options: {
    commentMarker: 'process'
  },
  dist: {
    files: [
      {
        expand: true,
        cwd: '<%= yeoman.dist %>',
        src: ['*.html', 'views/{,*/}*.html'],
        dest: '<%= yeoman.dist %>'
      }
    ]
  }
}
grunt.registerTask('build', [
// ...
'processhtml',
'usemin',
'htmlmin'
]);

3) Change way the application gets bootstrapped

a) remove ng-app="mockedBackendWithAngularjsApp" from the body tag in the index.html

b) add the code to your app.js to bootstrap the regular app

/**
 * @ngdoc bootstrap
 * @name mockedBackendWithAngularjsApp
 *
 */
(function () {

  if (!angular.mock) {
    angular.element(document).ready(function () {
      angular.bootstrap(document, ['mockedBackendWithAngularjsApp']);
    });
  }
})();

When you call grunt serve afterwards the app should still start.

c) create app-mock.js

angular
  .module('mockedBackendWithAngularjsAppDev', ['mockedBackendWithAngularjsApp', 'ngMockE2E'])
  .run(function ($httpBackend) {
    'use strict';
    $httpBackend.whenGET(/^views\//).passThrough();
    $httpBackend.whenGET(/^res\//).passThrough();

    /* backend API calls here */
    $httpBackend.whenPOST(/^\/signup/).respond(200);
    $httpBackend.whenGET(/^\/api\/catalog\/US/).respond(200, TD.catalogUS);
    $httpBackend.whenPOST(/\/api\/\/exception\/(\S)*/).respond({});

  });
if (angular.mock) {
  angular.element(document).ready(function () {
    'use strict';
    angular.bootstrap(document, ['mockedBackendWithAngularjsAppDev']);
  });
}

d) add your mock files to ./test/mock, e.g.,

  window.TD = window.TD || {};
  TD.catalogUS = {
    key: 'Hello World!'
  };

e) make mock files available – add folder to the livereload task in your Gruntfile

connect().use(
  '/mock',
  connect.static('./test/mock')
),

f) wire everything together – add files to index.html

4) Implement your backend calls with $http as usual. You can find an example in the main.js controller in the demo project.

Enjoy your new offline development environment!

These two days in Istanbul were quite impressive but at the same time the weather was so unbelievable rainy that I couldn’t visit much of the city. But beside that the conference about JS was worth to come – so let’s start with Saturday.

Saturday

It was a warm welcome and the registration process was well organized so I felt myself comfortable from the first minute on. The advertised breakfast wasn’t that good but we were not traveled so far to get a good dish at the conference so I shouldn’t bother.

The first talk with the title “Javascript <3 Unicode” by Mathias Bynens sounded a bit boring but it wasn’t at all. He pointed out all the pitfalls you will face in a daily business and this in a very entertaining talk. For example he explained the unicode plates, why a string reverse is not as simple as "foo".split('').reverse().join('') (try it with the unicode ‘pile of poo’) and mentioned quotes of celebrities to find the right solution. As general rule of thumb I will check the behavior of every user facing text input field regarding the unicode sign ‘pile of poo’ in the future. If this doesn’t break the app that it’s a good sign. And in addition I should mention that he pointed out even problems of MySQL (MySQL cut a string when it’s not configured in UTF-8 embec (something…) mode) and unicode what was very surprising to me.

The second talk was about “This is bigger than us: Building a future for Open Source” by Lena Reinhard. She did a great job by presenting such a complex and abstract theme at a JS conference in a simple an practical way. One important thing I would like to repeat is that software development needs more diversity. I guess this is so true especially since she also pointed out that only less then one percent of the world population are software engineers. So at one developer there are 399 not developers… That’s crazy when you look around and you find so many devices next to you in every second of your life.

The last talk before lunch was about “Scaling Node.js Applications with Redis, RabbitMQ and cote.js” by Armağan Amcalar. Because I’m personally a fan of Microservices I was super interested in this talk to hear about how others are trying to achieve this architecture goal. He talked about decoupling of node.js applications through an event driven development style across node.js applications and how this can be achieved by using for example pup/sub of Redis or Messages of RabbitMQ. In addition he mentioned cote.js – a library written by them which seems to be a more lite weight tool. But he couldn’t talk too much about it so everybody should take a look and build her/his own meaning.

The lunch break was big enough to even walk a bit around but sadly it was still such a bad weather that I couldn’t go around much. That’s why I will directly continue with the talk about “Getting Started with ClojureScript” by Üstün Özgür. He was pointing out all the good parts of functional programming and I have to agree with him especially because I wrote three years ago also an article printed in a magazine about functional programming methods in web development. The only thing I don’t agree is that ClojureScript is a better language then plain JS. In my opinion you can do already all the nice things directly with JS so I don’t see the point of compiling – I also haven’t seen it for CoffeeScript or GWT. The only exception is ASM.JS but this is a different topic so I will not go deeper here.

At 3pm “Meteor for Everyone” by Barış Güler started. It was nice to see more about this framework and what it is actually for. For me it really looks like a perfect thing for prototyping or tools with not that much expected load. I would probably use it for company internal tools for example. What I dislike is that they also invited there own ecosystem by explicitly not using NPM. Maybe it makes them faster but it’s even harder to get something like this running for production. Nevertheless the talk was enlightening.

“AngularJS Directives for D3JS” by Yaprak Ayazoğlu was the one talk I would have expected a bit more. She showed how you can encapsulate complicated d3.js statements within an AngularJS directive but this was it basically. Probably enough for people haven’t worked with AngularJS or d3.js previously but for me there wasn’t much new things.

The next talk about “Realtime MVC with Sails.js” by Serdar Doğruyol started pretty good by telling us about how easy it is to setup applications but the downside was that he didn’t shared with us any live session. I was expecting that I could at least see how easy it is to setup a hello world app. Maybe next time…

The talk “Ember.js Framework” by Sean Yu was more about “Why we have chosen Ember.js” than about the Framework itself. So when we rename it in our minds it was very good to hear about the decision making process of other companies how they decide which framework fit to them best. When I think about it’s one of the most important decisions you can make because probably a lot of developer will hate or love you even years after you made the decision. Beside this personal thing it can also lead to a lot of future development expense if you have made the wrong decision. That’s why I was happy to get this insights.

The last talk of the first day was “Hardware Development for JavaScript Developers” by Tarık Keleştemur. He did a great job presenting a way how to use JavaScript and an arduino. Even some small practical examples he showed us on stage what was really a cool thing.

The evening I shared with friends but it was still rainy and very windy…

Sunday

Actually I have to admit that we came a bit late on the second day and just hopped into the conference room when the first talk “Scaling TweetDeck’s frontend” by Andy Hume started. Thank you for sharing how you guys at Twitter facing all the different kinds of problems. He mentioned that they have a lot of legacy code but trying to deal with it and get all new things done with there own framework called “Flight.JS”. Also to hear about how modern a specification and documentation process can look like in such a big company gives me hope that it doesn’t have to be always too bureaucratic.

Pascal Precht’s talk about “Componentize all the things!” was super informative and gave the audience an idea how shiny the future of web development will look like. He covered to much in his talk that want start writing about all the things in this blog post but I highly recommend to check out his slides by yourself.

The last talk I could listen to was “It’s never to late to fight your legacy” by Mate Nadasdi (after that I had to leave to get my flight). This talk was a very nice addition to the Tweetdeck one but was diving deeper into the technical details about which tools can be helpful to move on and get out of the rotten code base. He mentioned tools like ESlint, Unit testing frameworks like sinon.js and so forth. It’s definitely worth for every JS developer to look at the slides and check the list of tools – it’s likely that you will find some very useful ones.

All in all it was an awesome conference and I hope there will be another one next year. Keep your good work and thank you to everybody who made this conference possible.