Linking of docker containers and further issues with buildbo_MySQL

Docker

All docker containers now exist, and one of the only things left to do is get all the containers playing nice with one another.

MySQL-app

I set out to breakout mysql into its' own docker containerand made good progress, but before proceeding further with debugging some setup problems, I checked out if anyone was opposed to using another mysql docker container as a foundation for our own. There are hundreds of mysql docker containers out there so it seemed silly to dupliate work if unnecessary. Noone had objections, so I went ahead and picked out a mysql docker container to use. I choseorchardup/mysqlfrom the Docker Index because it was pretty barebones and for the nice additional features it add in the form of being able to set environment variables in the container at runtime to do things like setup your own usernames, passwords, databases, etc, etc.

After awhile of trying to modify the run scripts that the orchardup/mysql image uses to launch the mysql server, I decided to back down for the time being. I was attempting to use orchardup/mysql as a base for our own mysql-app, so that I could then have our app do the additional loading of statusdb and scheduelrdb schemas. This proved to be a pain, and so rather than fight it further, I went with the redundant option of having buildapi-app and buildbot-app each individually load the schemas they needed into the database, regardless of if the schema already existed. I am not happy with this as a permanent solution for this development setup, but it should work well for our initial setup.

This also means that vagrant will now simply just need to pull the orchardup/mysql image, run it, forward ports, and link it with the other container apps, making this the lightest setup.

I modified buildbot-app and buildapi-app to use the newly created environment variables for the mysql app when connecting and using the databases (they appear upon running the docker containers when linking).

Buildbot-app

When I went to test buildbot-app, I ran into a an exceptions.ValueError: Malformed url

(Buildbot)root@96fbd42254f3:/# /start_buildbot.sh
mysql: option '-h' requires an argument
cd master && buildbot start $PWD
Following twistd.log until startup finished..
/usr/local/lib/python2.7/dist-packages/buildbot-0.8.2_hg_f6d9311d9246_production_0.8-py2.7.egg/buildbot/scripts/logwatcher.py:52: PotentialZombieWarning: spawnProcess called, but the SIGCHLD handler is not installed. This probably means you have not yet called reactor.run, or called reactor.run(installSignalHandler=0). You will probably never see this process finish, and it may become a zombie process.
  env=os.environ,
2014-05-21 05:35:52+0000 [-] Log opened.
2014-05-21 05:35:52+0000 [-] twistd 9.0.0 (/usr/bin/python2.7 2.7.3) starting up.
2014-05-21 05:35:52+0000 [-] reactor class: twisted.internet.selectreactor.SelectReactor.
2014-05-21 05:35:52+0000 [-] monkeypatch_twisted_cbLogin applied
2014-05-21 05:35:52+0000 [-] Creating BuildMaster — buildbot.version: 0.8.2-hg-f6d9311d9246-production-0.8
2014-05-21 05:35:52+0000 [-] loading configuration from /Buildbot/build-master/master.cfg
2014-05-21 05:35:52+0000 [-] unable to import dnotify, so Maildir will use polling instead
2014-05-21 05:35:52+0000 [-] JacuzziAllocator 44938192: created
2014-05-21 05:35:52+0000 [-] nextAWSSlave: start
2014-05-21 05:35:52+0000 [-] nextAWSSlave: start
2014-05-21 05:35:54+0000 [-] JacuzziAllocator 37763792: created
2014-05-21 05:35:54+0000 [-] nextAWSSlave: start
2014-05-21 05:35:54+0000 [-] nextAWSSlave: start
2014-05-21 05:35:59+0000 [-] finished loading config file
2014-05-21 05:36:01+0000 [-] BuildMaster listening on port tcp:9000
2014-05-21 05:36:01+0000 [-] configuration update started
2014-05-21 05:36:01+0000 [-] configuration update failed
2014-05-21 05:36:01+0000 [-] Unhandled Error
    Traceback (most recent call last):
      File "/usr/local/lib/python2.7/dist-packages/buildbot-0.8.2_hg_f6d9311d9246_production_0.8-py2.7.egg/buildbot/master.py", line 628, in loadTheConfigFile
        d = self.loadConfig(f)
      File "/usr/local/lib/python2.7/dist-packages/buildbot-0.8.2_hg_f6d9311d9246_production_0.8-py2.7.egg/buildbot/master.py", line 933, in loadConfig
        d.addCallback(lambda res:
      File "/usr/local/lib/python2.7/dist-packages/Twisted-9.0.0-py2.7-linux-x86_64.egg/twisted/internet/defer.py", line 190, in addCallback
        callbackKeywords=kw)
      File "/usr/local/lib/python2.7/dist-packages/Twisted-9.0.0-py2.7-linux-x86_64.egg/twisted/internet/defer.py", line 181, in addCallbacks
        self._runCallbacks()
    — —
      File "/usr/local/lib/python2.7/dist-packages/Twisted-9.0.0-py2.7-linux-x86_64.egg/twisted/internet/defer.py", line 323, in _runCallbacks
        self.result = callback(self.result, *args, **kw)
      File "/usr/local/lib/python2.7/dist-packages/buildbot-0.8.2_hg_f6d9311d9246_production_0.8-py2.7.egg/buildbot/master.py", line 934, in
        self.loadConfig_Database(db_url, db_poll_interval))
      File "/usr/local/lib/python2.7/dist-packages/buildbot-0.8.2_hg_f6d9311d9246_production_0.8-py2.7.egg/buildbot/master.py", line 1055, in loadConfig_Database
        db_spec = DBSpec.from_url(db_url, self.basedir)
      File "/usr/local/lib/python2.7/dist-packages/buildbot-0.8.2_hg_f6d9311d9246_production_0.8-py2.7.egg/buildbot/db/dbspec.py", line 175, in from_url
        raise ValueError("Malformed url")
    exceptions.ValueError: Malformed url
    

The buildmaster took more than 10 seconds to start, so we were unable to
confirm that it started correctly. Please 'tail twistd.log' and look for a
line that says 'configuration update complete' to verify correct startup.

make: *** [start] Error 1

It's possible this has to do with the mysql setup, as I possibly didn't link things up fully. More testing is necessary for tomorrow.

Buildapi-app

To run rabbitmq, mysql, and buildapi all linked together, run these commands in sequence

• docker run -d -p 5672:5672 -p 15672:15672 -p 4369:4369 -name rabbitmq rabbitmq-app
• docker run -d -p 3306:3306 -name=mysql orchardup/mysql
• docker run -t -i -p 8888:8888 -link rabbitmq:mq -link mysql:sql -name buildapi buildapi-app /bin/bash

This will drop you into a bash shell session in buildapi-app

When I attempt to run /start_selfserve_buildapi.sh I receive the following error:

root@e141d055c1c7:/# ./start_selfserve_buildapi.sh
Starting subprocess with file monitor
Running reloading file monitor
2014-05-21 06:37:13,352 Kombu connection revived
2014-05-21 06:37:13,353 Connected to amqp://selfserveagent@172.17.0.2:5672//
Traceback (most recent call last):
  File "/usr/local/bin/paster", line 9, in
    load_entry_point('PasteScript==1.7.3', 'console_scripts', 'paster')()
  File "/usr/local/lib/python2.7/dist-packages/paste/script/command.py", line 84, in run
    invoke(command, command_name, options, args[1:])
  File "/usr/local/lib/python2.7/dist-packages/paste/script/command.py", line 123, in invoke
    exit_code = runner.run(args)
  File "/usr/local/lib/python2.7/dist-packages/paste/script/command.py", line 218, in run
    result = self.command()
  File "/usr/local/lib/python2.7/dist-packages/paste/script/serve.py", line 276, in command
    relative_to=base, global_conf=vars)
  File "/usr/local/lib/python2.7/dist-packages/paste/script/serve.py", line 313, in loadapp
    **kw)
  File "/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 247, in loadapp
    return loadobj(APP, uri, name=name, **kw)
  File "/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 272, in loadobj
    return context.create()
  File "/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 710, in create
    return self.object_type.invoke(self)
  File "/usr/local/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py", line 146, in invoke
    return fix_call(context.object, context.global_conf, **context.local_conf)
  File "/usr/local/lib/python2.7/dist-packages/paste/deploy/util.py", line 56, in fix_call
    val = callable(*args, **kw)
  File "/buildapi/buildapi/config/middleware.py", line 55, in make_app
    config = load_environment(global_conf, app_conf)
  File "/buildapi/buildapi/config/environment.py", line 66, in load_environment
    init_scheduler_model(scheduler_engine)
  File "/buildapi/buildapi/model/__init__.py", line 7, in init_scheduler_model
    scheduler_db_meta.reflect(bind=engine)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/schema.py", line 2342, in reflect
    conn = bind.contextual_connect()
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2284, in contextual_connect
    self.pool.connect(),
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 209, in connect
    return _ConnectionFairy(self).checkout()
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 370, in __init__
    rec = self._connection_record = pool._do_get()
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 696, in _do_get
    con = self._create_connection()
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 174, in _create_connection
    return _ConnectionRecord(self)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 255, in __init__
    self.connection = self.__connect()
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 315, in __connect
    connection = self.__pool._creator()
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/strategies.py", line 80, in connect
    return dialect.connect(*cargs, **cparams)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 275, in connect
    return self.dbapi.connect(*cargs, **cparams)
  File "/usr/local/lib/python2.7/dist-packages/MySQLdb/__init__.py", line 81, in Connect
    return Connection(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/MySQLdb/connections.py", line 187, in __init__
    super(Connection, self).__init__(*args, **kwargs2)
sqlalchemy.exc.OperationalError: (OperationalError) (2005, "Unknown MySQL server host 'SQL_PORT_3306_TCP_ADDR' (0)") None None

Again, this looks like a linking issue and more testing is necessary.

What's next?

The next steps are these:

• Resolve exceptions.ValueError in buildbot-app
• Resolve sqlalchemy.exc.OperationalError in buildapi-app
• Link rabbitmq, mysql, and buildapi and test that everything works
• Link mysql, and buildbot and test that everything works
• Link rabbitmq, mysql, buildapi AND buildbot and test that the whole package works
• See if there is a good way to load statusdb and schedulerdb schemas into mysql in a mysql-app setup built on the orchardup/mysql image. This would prevent the redundanc of loading schemas in buildapi-app and buildbot-app

Things I found useful here

• docker logs
• vboxmanage modifyvm boot2docker-vm –-nic1 delete http
• vboxmanage modifyvm boot2docker-vm –natpf1 "http,tcp,127.0.0.1,8888,,8888"

Things to look into

• VOLUME docker command
• Renaming apps as mozilla/buildbot-dev or mozilla/buildbot-dev
• Setting multiple natpf's for boot2docker testing