Brief OpenShift troubleshooting

If you have issues after an automagic openshift-on-openstack deployment:

1. Remember: every buildconfig created *before* the registry is not authorized to push the images

2. Remember: hawkular is a java application. Startup is slow. Just click there and wait for the startup

3. Ansible is your friend. To get container logs, just

ansible all -m shell -a 'ls /var/log/containers/CONTAINER_NAME*'

ansible all -m shell -a 'cat /var/log/containers/CONTAINER_NAME*' > CONTAINER_NAME.log

4. If a container don’t startup during the deployment, a broken image may have been downloaded

Jun 1 23:30:36 dev-7-infra-0 atomic-openshift-node: I0601 23:30:36.234103 32913 server.go:608] Event(api.ObjectReference{Kind:"Pod", Namespace:"default", Name:"router-1-deploy", UID:"033670a9-470e-11e7-878f-fa163eac2bf7", APIVersion:"v1", ResourceVersion:"936", FieldPath:""}): type: 'Warning' reason: 'FailedSync' Error syncing pod, skipping: failed to "StartContainer" for "POD" with RunContainerError: "runContainer: Error response from daemon: {\"message\":\"invalid header field value \\\"oci runtime error: container_linux.go:247: starting container process caused \\\\\\\"exec: \\\\\\\\\\\\\\\"/pod\\\\\\\\\\\\\\\": stat /pod: no such file or directory\\\\\\\"\\\\n\\\"\"}"

Cleanup docker repo

docker ps -aq | xargs docker rm
docker rmi 90e9207f44f0 --force

5. Run oadm diagnostics on the master 😉

6. Check #oc get hostsubnet

Adding docker images to openshift 3.1

Openshift 3.1 is based on Kubernetes and Docker, and provides a small set of images including jboss EAP 6.4.

You can add new images in two steps:

1- create an ImageStream, that’s a docker image + a set of labels
2- create a Template using that ImageStream

To create the ImageStream read carefully the following description.

# Create the ImageStream
oc create -f - <<EOF
apiVersion: v1
kind: ImageStream
  name: wildfly9-openshift
  namespace: openshift        # Set this to "openshift" if you want to make this image globally visible
  dockerImageRepository:  # The original docker hub repo
  - annotations:
      description: Wildfly 9.0 S2I images.
      iconClass: icon-jboss
      sampleRef: 9.0.x 
      supports: wildfly:9,javaee:7,java:8,
      tags: builder,javaee,java,jboss
      version: "1.0"
    name: "1.0"
  dockerImageRepository: ""

docker multihost network: an epiphany of namespaces.

Playing with docker multihost network this week-end.

With multihost networking you can run communicating containers on different docker nodes.
The magic relies on:
– a shared kv store (Eg. consul) for ipaddresses;
– a netns for vxlan for communication with a bridge and no processes attached.

Every network created using the Overlay driver has its own network namespace.
And for every network (& its subnet combination), we create a linux bridge inside that dedicated namespace.
The host end of the veth pair is moved into this namespace and attached to the bridge (inside of that namespace).
Hence, if you look for the veth pair in the host namespace, you wont find any :-).

If you look for vxlan setup on the boot2docker distro you have to dig deep ;).
1- docker netns is stored in /var/run/docker/netns. To access it you need to

#ln -s /var/run/docker/netns /var/run;

2- Now you can look for the vxlan netns, which has the same id on every machine:

#ip netns ls | while read a; do
    ip netns exec $a ip l | grep vxlan -q && echo $a;done

The vxlan references the UDP port for communication (eg. dstport 46354).

87: vxlan1:  mtu 1500 qdisc noqueue master br0 state UNKNOWN mode DEFAULT group default
    link/ether da:69:8d:4d:b9:39 brd ff:ff:ff:ff:ff:ff promiscuity 1
    vxlan id 256 srcport 0 0 dstport 46354 proxy l2miss l3miss ageing 300

3- Every container with EXPOSEd ports has a veth paired with a veth in the vxlan netns;

4- the veth in vxlan netns are slaves of br0;

5- br0 has an ip, and is the default gw for containers.

NetworkManager please, stay away from my docker0

To set a list of unmanaged-devices you can just do the following.

cat >> /etc/NetworkManager/NetworkManager.conf <<EOF




sudo nmcli connection reload

Strangely I had to put this in NetworkManager.conf. Using
/etc/NetworkManager/conf.d/20-unmanaged-bridges.conf didn’t work.

Access your docker volumes via sftp!

Docker DNS is a python application you can use to
identify your containers by hostname or name.

You can run it with behind dnsmasq using the instructions

#sudo twistd dockerdns -u http://docker-server:5000 -p 53
# host mycontainer.docker # by container-name or hostname
# host mymage.*.docker # all container from a given image

Using twisted conch, I added sftp functionalities too, so you can access your container volumes directly.

#sftp -P10022 mycontainer@docker-server
sftp$ ls /

Eat the (now is docker-compose)

Fig is a very simple Docker orchestrator. It can be used to describe your container environment and make it easy to replicate.
cat fig.yml
# Setup two linked containers: jboss and a log server (you need to configure
# wildfly to use syslog)
  image: jboss/wildfly
    - JAVA_OPTS=  " -Xdebug -Xrunjdwp:server=y,transport=dt_socket,address=4000,suspend=n "
    - "8080:8080"
    - syslogserver

    image:  jplock/rsyslog
       -  /var/log:/var/log