Integration in GitLab Runner CI/CD

Hello everyone,
we are currently developing our ci/cd pipeline with a GitLab Runner. In this pipeline we want to build a web-application which has access to ActyxOS. In order to automate this progress we have to create an Actyx Docker container in our pipeline. Usually you would be able to create a simple Docker container by writing the image-name into the gitlab configuration-file. But in the configuration file you are no able to set extra configuration for port mapping or volumes. So I am not able to create that Actyx container the right way. Do you have any experience with GitLab Runner? Or do you might have any solution for my problem?

Thank you,
Michel

Welcome @michel! :wave:

TL;DR: I managed to start a GitLab runner in a way that starts an Actyx node, but it feels a bit hacky. Still working on allowing a container to use that node, though.
Probably wrapping your app into an image containing Actyx and running it as background process is a better approach.

EDIT: Added PoC for wrapping your app + Actyx into a single image
EDIT: Add workaround for data directory mount.


Simply including actyx/os as a service in GitLab CI failed with the following errors:

2021-03-30T16:52:21.958924178Z ip: RTNETLINK answers: Operation not permitted
2021-03-30T16:52:21.960238952Z Cannot start ActyxOS due to the following errors:
2021-03-30T16:52:21.960253243Z     The container is not running in --privileged mode

2021-03-30T16:52:21.960257421Z     /data does not exist or is mounted readonly. Please attach a persistent volume to that position, e.g. '-v /mnt/home/actyx/:/data'. Check the docs for further information.
2021-03-30T16:52:21.960261853Z     /data is correctly mounted in rw mode, but doesn't look writeable.

So it seems we have two issues:

  1. Actyx needs to be able to access the network / open ports
  2. We need a writeable data volume

Separate images

1. Network

For the actyx/os image to be able to talk to the hosts network, it needs to run in privileged mode. I started my GitLab CI runner w/ --cap-add=NET_ADMIN and --privileged.

$ docker run -d --name gitlab-runner --restart always --privileged --cap-add=NET_ADMIN \    
    -v /var/run/docker.sock:/var/run/docker.sock \
    -v gitlab-runner-config:/etc/gitlab-runner \
    gitlab/gitlab-runner:latest

When registering the runner w/ GitLab, you also need to set the privileged mode using --docker-privileged, e.g. docker run --rm -it --privileged -v gitlab-runner-config:/etc/gitlab-runner gitlab/gitlab-runner:latest register --docker-privileged

After these two steps, the Actyx the RTNETLINK answers: Operation not permitted is gone, but it still complains about the data volume.

2. Data volume

The start script in the actyx/os docker image checks /etc/mtab to make sure a data volume is mounted. That makes sense, as it prevents users from accidentally using ephemeral volumes; thus from loosing data.
When running in CI, ephemeral volumes are what we want, normally. So we need to trick Actyx in the container into accepting a local folder as data volume. We can do this by overriding the entrypoint/command to create a local folder and mount it locally so it shows up in /etc/mtab:

docker run -it -eAX_DEV_MODE=1 --privileged  \
  --entrypoint="" \
   actyx/os:1.1.2 \
   sh -c "mkdir /data && mkdir -p /tmp/actyxos && mount --bind /tmp/actyxos /data && /tmp/init.sh"

This can be replicated in .gitlab-ci.yml, see below.

After doing so, the Actyx node starts up successfully:

(probably someone from engineering is already headed to my place to ask why I’m suggesting weird hacks show me an elegant solution)

Config

Here’s the complete .gitlab-ci.yml file

image: curlimages/curl

stages:
  - test

actyx:ping:
  stage: test

  variables:
    AX_DEV_MODE: 1

  services:
    - name: actyx/os
      alias: actyx
      entrypoint: [""]
      command: ["sh", "-c", "mkdir /data && mkdir -p /tmp/actyxos && mount --bind /tmp/actyxos /data && /tmp/init.sh"]
  script:
    curl localhost:4457/_internal/swarm/state

Actyx + App in single image

Alternatively, and probably more elegantly, we can run Actyx in the background in our application image. We’d use supervisord or similar for that.

Here’s a simple PoC that shows how this would work:

  • We build our own docker image based on actyx/os and add our application
  • We call the original image’s entrypoint to run in the background and start our app
  • Then we can use this image directly in GitLab as all communication between our app and Actyx happens locally

app.sh just tries to connect to Actyx and outputs the state of the Actyx swarm:

#!/bin/sh

nc -z localhost:8080 && echo Connected to actyx node
echo Swarm state:
curl localhost:4457/_internal/swarm/state

entrypoint.sh prepares the data folder as shown in the previous section, starts Actyx and our app afterwards

#!/bin/sh

# Make actyx accept local folders as data volumne
mkdir /data
mkdir -p /tmp/actyxos
mount --bind /tmp/actyxos /data 

# Start actyx node and wait for it (naively)
/tmp/init.sh & # that's the original image's entrypoint
sleep 15s 

/app.sh

Finally, our Dockerfile ties it together:

FROM actyx/os:1.1.2

ADD entrypoint.sh /entrypoint.sh
ADD app.sh /app.sh

ENTRYPOINT ["/entrypoint.sh"]

Building and running this image should result in this:


$ docker build . -t wrapper-poc && docker run -it --privileged wrapper-poc  

 => [internal] load build definition from Dockerfile                                                                                                                                                                               0.0s
 => => transferring dockerfile: 36B                                                                                                                                                                                                0.0s
 => [internal] load .dockerignore                                                                                                                                                                                                  0.0s
 => => transferring context: 2B                                                                                                                                                                                                    0.0s
 => [internal] load metadata for docker.io/actyx/os:1.1.2                                                                                                                                                                          0.0s
 => [1/3] FROM docker.io/actyx/os:1.1.2                                                                                                                                                                                            0.0s
 => [internal] load build context                                                                                                                                                                                                  0.0s
 => => transferring context: 60B                                                                                                                                                                                                   0.0s
 => CACHED [2/3] ADD entrypoint.sh /entrypoint.sh                                                                                                                                                                                  0.0s
 => CACHED [3/3] ADD app.sh /app.sh                                                                                                                                                                                                0.0s
 => exporting to image                                                                                                                                                                                                             0.0s
 => => exporting layers                                                                                                                                                                                                            0.0s
 => => writing image sha256:bdd6c33d2aa8e2fc29e05943a56af00a533e7c0929f6748d4fdc7ed8639a965c                                                                                                                                       0.0s
 => => naming to docker.io/library/wrapper-poc                                                                                                                                                                                     0.0s
Starting ActyxOS
***********************
**** ActyxOS ready ****
***********************
Connected to actyx node
Swarm state:
{"Ok":{"store":{"count":0,"size":0},"swarm":{"listen_addrs":["/ip4/127.0.0.1/tcp/4001","/ip4/172.17.0.3/tcp/4001","/ip4/172.26.0.1/tcp/4001"],"peer_id":"12D3KooW9u4JS9dFhSmujV1yZczuKxwUd3D9kTyFmBAFsYSXVvDL","peers":{}}}}%


References:

The approach @wwerner outlined is the way to go if you want to do integration tests with ActyxOS on Docker, meaning you want to employ the Docker Runtime that comes along with it (ActyxOS on Docker is running Docker-in-Docker, that’s the reason for the privileged requirement).

However, if you just want to run ActyxOS and make its services accessible to either the host or to another container, you can create a small Dockerfile yourself wrapping actyxos-linux:

FROM ubuntu:20.04

RUN apt-get update && apt-get upgrade -y
RUN apt-get install -y wget

# Last available version from https://downloads.actyx.com/
RUN wget https://axartifacts.blob.core.windows.net/artifacts/feea217d729d06224838c4d2fcf35ba51c87c64e/linux-binaries/linux-x86_64/actyxos-linux \
    && mv actyxos-linux /usr/bin/ && chmod +x /usr/bin/actyxos-linux

# Pond WS API
EXPOSE 4243/tcp
# Swarm Port
EXPOSE 4001/tcp
# Event Service API
EXPOSE 4454/tcp
# Admin API
EXPOSE 4457/tcp
# Admin API #2
EXPOSE 4458/tcp
# Can be ephemeral or mapped from the host
VOLUME /data
# Event Service API must bind to all interfaces when running inside Docker
CMD ["actyxos-linux", "--working_dir", "/data", "0.0.0.0"]

You can build that with docker build -t ci-runner . and use it like so:

mkdir data && docker run --rm -v `pwd`/data:/data -p 4454:4454 ci-runner

And in another shell, query the Event Service API:

➜ ~ curl localhost:4454/api/v1/events/offsets     
{}

The other ports can be exposed depending on your requirements.


This way, you can even prepopulate the /data directory with data suitable for your test setup!

I hope I could help you out; would love to hear more on how you plan to do integration testing with a web app! Is it orchestrated with something like Selenium or Puppet?

Regards,
Oliver

1 Like

First of all, thank you really much for your quick answers. Unfortunately running the GitLab Runner in privileged mode is not an option. I know without the privileged mode Docker-in-Docker is not possible.

You might heard about Kaniko, the alternative to build Docker Images in a Gitlab Runner. I tried to build my own Actyx-Linux Image with the Dockerfile @wngr mentioned above. This worked successfully. Then I’m trying to link that new Image as a Service in my GitLab Runner configuration, which is also working. Unfortunately I am still not able to access the Actyx Service. Did you try any approach with Kaniko? If yes did you have success?

Regards,
Michel

Hey @michel,

our pleasure, just not sure I fully understand your use case yet.
Could you perhaps explain a bit further what you’re trying to achieve?
Do you want to, e.g., run integration tests against an application running on one (or several) nodes? Do you want to build/package a web app?

Thanks,
Wolfgang

Hi @wwerner ,

I am trying do my integration tests inside the pipeline. When the tests are successful, I would like to build the image of the webapp (without Actyx). The last past is not a problem, this is already working. But I’m not able to access the Actyx node in the pipeline at the moment.

Thank you,
Michel

Hello everyone,
I was able to use some parts of your answers and built my own solution. In my pipeline I’m starting with building my own Actyx image. This image is first uploaded to the GitLab container registry and then used as a service in my webapp. For the Actyx Image I used the Dockerfile @wngr mentioned in his answer:

FROM ubuntu:20.04
RUN sudo apt-get update && apt-get upgrade -y
RUN sudo apt-get install -y wget

# Last available version from https://downloads.actyx.com/
RUN wget https://axartifacts.blob.core.windows.net/artifacts/feea217d729d06224838c4d2fcf35ba51c87c64e/linux-binaries/linux-x86_64/actyxos-linux \
    && mv actyxos-linux /usr/bin/ && chmod +x /usr/bin/actyxos-linux

# Pond WS API
EXPOSE 4243/tcp
# Swarm Port
EXPOSE 4001/tcp
# Event Service API
EXPOSE 4454/tcp
# Admin API
EXPOSE 4457/tcp
# Admin API #2
EXPOSE 4458/tcp
# Can be ephemeral or mapped from the host
VOLUME /data
# Event Service API must bind to all interfaces when running inside Docker
CMD ["actyxos-linux", "--working_dir", "/data", "0.0.0.0"]

With this Image I am able to use Actyx in Docker without the privileged mode but with access to the correct ports and the data volume. My GitLab Runner Configuration looks like this:

stages:
  - prebuild
  - test
  - build
  - deploy

build_actyx:
  stage: prebuild
  image:
    name: gcr.io/kaniko-project/executor:debug
    entrypoint: [""]
  variables:
    KUBERNETES_MEMORY_REQUEST: 2Gi
    KUBERNETES_MEMORY_LIMIT: 2Gi
  script:
    - mkdir -p /kaniko/.docker
    - echo "{\"auths\":{\"$CI_REGISTRY\":{\"username\":\"$CI_REGISTRY_USER\",\"password\":\"$CI_REGISTRY_PASSWORD\"}}}" > /kaniko/.docker/config.json
    - /kaniko/executor --context $CI_PROJECT_DIR --dockerfile $CI_PROJECT_DIR/docker/actyx/Dockerfile --destination $CI_REGISTRY_IMAGE --snapshotMode=redo --cleanup

test_application:
  stage: test
  image: node:latest
  services:
    - gitlab.xxxxxxx.xx:xxxx/actyx/actyx-template-cordova:latest
  before_script:
    - npm i -q --no-warnings
    - npm i -qg cordova
    - mkdir www
    - cordova platform add electron@2.0.0
  script: 
    - curl localhost:4454/api/v1/events/offsets   
    - npm run test:once
  when: on_success 

As you can see, I am first building the Actyx image with Kaniko and then I am pushing it to our GitLab Container Registry. Then I am using the new Actyx image from our registry as a service. This is working fine for tests. When all tests are done I am able to build my WebApp without Actyx.

3 Likes