Add experimental Debian support.

Note, there is something weird with those images, as the will segfault on the first run (I've checked that with latest Debian 11 images, maybe it will change in the future), so there is forced reboot after certain amount of time as a workaround.
Use predefined os types for vbox VMs.
2026-03-25 14:43:32 +01:00 · 2023-04-18 21:48:19 +02:00 · 2023-04-18 21:39:56 +02:00 · 2023-04-02 10:05:06 +02:00 · 2023-02-26 19:20:11 +01:00 · 2023-02-22 21:10:59 +01:00
6 changed files with 159 additions and 77 deletions
--- a/README.rst
+++ b/README.rst
@@ -2,8 +2,8 @@
 box.py
 ======

-Box.py is a simple automation tool meant to run Ubuntu, Fedora or Centos Stream
-cloud images on top of VirtualBox.
+Box.py is a simple automation tool meant to run Ubuntu, Fedora, Centos Stream
+or Debian cloud images on top of VirtualBox.

 What it does is simply download official cloud image, set up VM, tweak it up
 and do the initial pre-configuration using generated config drive.
@@ -16,7 +16,7 @@ weird named options for ``vboxmanage`` ;P)
 Requirements
 ------------

- Python >=3.7
+- Python >=3.8

  - `pyyaml`_
  - `requests`_
@@ -24,7 +24,7 @@ Requirements
 - Virtualbox (obviously)
 - ``mkisofs`` or ``genisoimage`` command for generating ISO image
 - ``wget`` command for fetching images
- ``sha256sum`` command for checksum check
+- ``sha256sum`` and ``sha512sum`` commands for checksum check
 - ``qemu-img`` from *qemu-utils* package command for converting between images
  formats

@@ -39,7 +39,7 @@ your operating system, or by using virtualenv for Python requirements, i.e.:

   $ python -m virtualenv .venv
   $ . .venv/bin/activate
-   (.venv) $ pip install requirements.txt
+   (.venv) $ pip install -r requirements.txt

 then you can issue:

@@ -91,13 +91,15 @@ use it ad-hoc, or place on your ``.bashrc`` or whatever:

 Currently, following commands are available:

- ``list`` - for quickly listing all/running VMs
- ``info`` - to get summary about VM
- ``destroy`` - that is probably obvious one
+- ``completion`` - as described above
 - ``create`` - create new VM
+- ``destroy`` - that is probably obvious one
+- ``info`` - to get summary about VM
+- ``list`` - for quickly listing all/running VMs
 - ``rebuild`` - recreate specified VM
 - ``ssh`` - connect to the VM using ssh
- ``completion`` - as described above
+- ``start`` - stop the running VM
+- ``stop`` - start stopped VM

 All of the commands have a range of options, and can be examined by using
 ``--help`` option.
--- a/box.py
+++ b/box.py
@@ -23,9 +23,10 @@ __version__ = "1.3"
 CACHE_DIR = os.environ.get('XDG_CACHE_HOME', os.path.expanduser('~/.cache'))
 CLOUD_IMAGE = "ci.iso"
 FEDORA_RELEASE_MAP = {'32': '1.6', '33': '1.2', '34': '1.2'}
+DEBIAN_CODENAME_MAP = {'12': 'bookworm', '11': 'bullseye', '10': 'buster'}
 TYPE_MAP = {'HardDisk': 'disk', 'DVD': 'dvd', 'Floppy': 'floppy'}
 DISTRO_MAP = {'ubuntu': 'Ubuntu', 'fedora': 'Fedora',
-              'centos': 'Centos Stream'}
+              'centos': 'Centos Stream', 'debian': 'Debian'}
 META_DATA_TPL = string.Template('''\
 instance-id: $instance_id
 local-hostname: $vmhostname
@@ -160,8 +161,8 @@ _boxpy() {
                        _ssh_identityfile
                        ;;
                    --distro)
-                        COMPREPLY=( $(compgen -W "ubuntu fedora centos" \
-                                -- ${cur}) )
+                        COMPREPLY=( $(compgen -W "ubuntu fedora centos
+                            debian" -- ${cur}) )
                        ;;
                    --type)
                        COMPREPLY=( $(compgen -W "gui headless sdl separate" \
@@ -585,6 +586,17 @@ class Config:
        return source


+class OsTypes:
+    def ubuntu(conf):
+        return "Ubuntu%s_64" % conf.version.replace('.', '_')
+
+    def fedora(conf):
+        return "Fedora_64"
+
+    def debian(conf):
+        return "Debian%s_64" % conf.version
+
+
 class VBoxManage:
    """
    Class for dealing with vboxmanage commands
@@ -765,7 +777,15 @@ class VBoxManage:
               '--acpi', 'on',
               '--audio', 'none',
               '--nic1', 'nat',
-               '--natpf1', f'boxpyssh,tcp,,{port},,22']
+               '--natpf1', f'boxpyssh,tcp,,{port},,22',
+               '--graphicscontroller', 'vmsvga',
+               '--vram', '16']
+
+        if hasattr(OsTypes, conf.distro):
+            cmd.extend(['--ostype', getattr(OsTypes, conf.distro)(conf)])
+        else:
+            cmd.extend(['--ostype', 'Linux_64'])
+
        for count, (hostport, vmport) in enumerate(conf.forwarding.items(),
                                                   start=1):
            cmd.extend(['--natpf1', f'custom-pf-{count},tcp,,{hostport},'
@@ -914,6 +934,7 @@ class VBoxManage:
 class Image:
    URL = ""
    IMG = ""
+    CHECKSUMTOOL = 'sha256sum'

    def __init__(self, vbox, version, arch, release, fname=None):
        self.vbox = vbox
@@ -964,7 +985,7 @@ class Image:
            return False

        if os.path.exists(os.path.join(CACHE_DIR, self._img_fname)):
-            cmd = ['sha256sum', os.path.join(CACHE_DIR, self._img_fname)]
+            cmd = [self.CHECKSUMTOOL, os.path.join(CACHE_DIR, self._img_fname)]
            calulated_sum = Run(cmd).stdout.split(' ')[0]
            LOG.details('Checksum for image: %s, expected: %s', calulated_sum,
                        expected_sum)
@@ -978,7 +999,8 @@ class Image:
            return True

        fname = os.path.join(CACHE_DIR, self._img_fname)
-        LOG.header('Downloading image %s', self._img_fname)
+        LOG.header('Downloading image %s from %s', self._img_fname,
+                   self._img_url)
        Run(['wget', '-q', self._img_url, '-O', fname])

        if not self._checksum():
@@ -1016,6 +1038,30 @@ class Ubuntu(Image):
        return expected_sum


+class Debian(Image):
+    URL = "https://cloud.debian.org/images/cloud/%s/daily/latest/%s"
+    IMG = "debian-%s-generic-%s-daily.qcow2"
+    CHECKSUMTOOL = 'sha512sum'
+
+    def __init__(self, vbox, version, arch, release, fname=None):
+        super().__init__(vbox, version, arch, release)
+        self._img_fname = self.IMG % (version, arch)
+        self._img_url = self.URL % (release, self._img_fname)
+        self._checksum_file = 'SHA512SUMS'
+        self._checksum_url = self.URL % (release, self._checksum_file)
+
+    def _get_checksum(self, fname):
+        expected_sum = None
+        Run(['wget', self._checksum_url, '-q', '-O', fname])
+        with open(fname) as fobj:
+            for line in fobj.readlines():
+                if self._img_fname in line:
+                    expected_sum = line.split(' ')[0]
+                    break
+
+        return expected_sum
+
+
 class Fedora(Image):
    URL = ("https://download.fedoraproject.org/pub/fedora/linux/releases/%s/"
           "Cloud/%s/images/%s")
@@ -1048,7 +1094,7 @@ class CentosStream(Image):
    IMG = '.*(CentOS-Stream-GenericCloud-%s-[0-9]+.[0-9].%s.qcow2).*'
    CHKS = "CHECKSUM"

-    def __init__(self, vbox, version, arch, release):
+    def __init__(self, vbox, version, arch, release, fname=None):
        super().__init__(vbox, version, arch, release)
        self._checksum_file = '%s-centos-stream-%s-%s' % (self.CHKS, version,
                                                          arch)
@@ -1104,7 +1150,7 @@ DISTROS = {'ubuntu': {'username': 'ubuntu',
                      'realname': 'ubuntu',
                      'img_class': Ubuntu,
                      'amd64': 'amd64',
-                      'default_version': '20.04'},
+                      'default_version': '22.04'},
           'fedora': {'username': 'fedora',
                      'realname': 'fedora',
                      'img_class': Fedora,
@@ -1114,13 +1160,20 @@ DISTROS = {'ubuntu': {'username': 'ubuntu',
                      'realname': 'centos',
                      'img_class': CentosStream,
                      'amd64': 'x86_64',
-                      'default_version': '8'}}
+                      'default_version': '8'},
+           'debian': {'username': 'debian',
+                      'realname': 'debian',
+                      'img_class': Debian,
+                      'amd64': 'amd64',
+                      'default_version': '11'}}


 def get_image_object(vbox, version, image='ubuntu', arch='amd64'):
    release = None
    if image == 'fedora':
        release = FEDORA_RELEASE_MAP[version]
+    if image == 'debian':
+        release = DEBIAN_CODENAME_MAP[version]
    return DISTROS[image]['img_class'](vbox, version, DISTROS[image]['amd64'],
                                       release, DISTROS[image].get('image'))

@@ -1246,6 +1299,7 @@ def vmcreate(args, conf=None):
           f'ssh://{username}@localhost:{vbox.vm_info["port"]}',
           'sudo cloud-init status']
    try:
+        counter = 0
        while True:
            out = Run(cmd)
            LOG.debug('Out: %s', out.stdout)
@@ -1265,6 +1319,15 @@ def vmcreate(args, conf=None):
                                          f'VM with ssh for user {username}. '
                                          f'Check output in debug mode.')
                time.sleep(3)
+                counter += 1
+                if counter == 8 and conf.distro == 'debian':
+                    counter += 1
+                    # there is something odd with debian cloud images, as they
+                    # segfault on first run. after ~20 seconds there should
+                    # already be panic, reset machine should help
+                    vbox.poweroff()
+                    time.sleep(3)
+                    vbox.poweron(args.type)
                continue

            LOG.info(' done')
@@ -1444,6 +1507,13 @@ def vmrebuild(args):
        conf.disk_size = vbox.get_media_size(disk_path)

    vmdestroy(args)
+
+    # Wait till VM is gone
+    while True:
+        vbox = VBoxManage(args.name)
+        if not vbox.get_vm_info():
+            break
+
    vmcreate(args, conf)
    return 0

@@ -1469,11 +1539,14 @@ def connect(args):
        return 16

    username = conf.username or DISTROS[conf.distro]["username"]
-    return Run(['ssh', '-o', 'StrictHostKeyChecking=no',
-                '-o', 'UserKnownHostsFile=/dev/null',
-                '-i', conf.ssh_key_path[:-4],
-                f'ssh://{username}'
-                f'@localhost:{conf.port}'], False).returncode
+    cmd = ['ssh', '-o', 'StrictHostKeyChecking=no',
+           '-o', 'UserKnownHostsFile=/dev/null',
+           '-i', conf.ssh_key_path[:-4],
+           f'ssh://{username}'
+           f'@localhost:{conf.port}']
+    LOG.debug('Connecting to vm `%s` using command:\n%s', args.name,
+              ' '.join(cmd))
+    return Run(cmd, False).returncode


 def _set_vmstate(name, state):
--- a/examples/devstack-ubuntu-20.04.yaml
+++ b/examples/devstack-ubuntu-20.04.yaml
@@ -29,7 +29,7 @@ runcmd:
  - [apt, purge, '-y', python3-pyasn1-modules]
  - [apt, purge, '-y', python3-simplejson]
  - [su, -, ubuntu, -c, "git clone https://github.com/gryf/vmstrap"]
-  - [su, -, ubuntu, -c, "vmstrap/bootstrap.sh"]
+  - [su, -, ubuntu, -c, "vmstrap/bootstrap.sh -c"]
  - [rm, -fr, /home/ubuntu/vmstrap]
  - [su, -, ubuntu, -c, "cp /tmp/local.conf /home/ubuntu/devstack/"]
  - [su, -, ubuntu, -c, "echo 'export HOST_IP=10.0.2.15' >> .bashrc"]
--- a/examples/kuryr_multinode_master.yaml
+++ b/examples/kuryr_multinode_master.yaml
@@ -34,55 +34,55 @@ write_files:
    content: |
      [[local|localrc]]
      disable_all_services
-      
+
      # Cinder
      disable_service c-api
      disable_service c-bak
      disable_service c-sch
      disable_service c-vol
      disable_service cinder
-      
+
      # Coredns
      disable_service coredns
-      
-      # Dstat 
+
+      # Dstat
      disable_service dstat
-      
+
      # ETCD
      enable_service etcd3
-      
+
      # Glance
      enable_service g-api
      enable_service g-reg
-      
+
      # Horizon
      disable_service horizon
-      
+
      # Keystone
      enable_service key
-      
+
      # kubernetes
      enable_service kubernetes-master
      disable_service kubernetes-worker
-      
+
      # kuryr
      enable_service kuryr-daemon
      enable_service kuryr-kubernetes
-      
+
      # mysql.
      enable_service mysql
-      
+
      # Nova services
      enable_service n-api
      enable_service n-api-meta
      enable_service n-cond
      enable_service n-cpu
      enable_service n-sch
-      
+
      # Neutron
      enable_service neutron
      enable_service neutron-tag-ports-during-bulk-creation
-      
+
      # Octavia
      enable_service o-api
      enable_service o-cw
@@ -90,41 +90,41 @@ write_files:
      enable_service o-hk
      enable_service o-hm
      enable_service octavia
-      
+
      # Neutron ovn services
      enable_service ovn-controller
      enable_service ovn-northd
      enable_service ovs-vswitchd
      enable_service ovsdb-server
-      
+
      # Placement API
      enable_service placement-api
      enable_service placement-client
-      
+
      # Neutron services
      enable_service q-ovn-metadata-agent
      enable_service q-qos
      enable_service q-svc
      enable_service q-trunk
      enable_service rabbit
-      
+
      # Swift
      disable_service s-account
      disable_service s-container
      disable_service s-object
      disable_service s-proxy
-      
+
      # Tempest
      enable_service tempest
-      
+
      # TLS
      disable_service tls-proxy
-      
+
      # Vars
      ADMIN_PASSWORD="secretadmin"
      DATABASE_PASSWORD="secretdatabase"
      ETCD_USE_RAMDISK="True"
-      
+
      KURYR_ENABLED_HANDLERS="vif,endpoints,service,namespace,pod_label,policy,kuryrnetworkpolicy,kuryrnetwork,kuryrport,kuryrloadbalancer"
      KURYR_ENFORCE_SG_RULES="False"
      KURYR_EP_DRIVER_OCTAVIA_PROVIDER="ovn"
@@ -134,30 +134,30 @@ write_files:
      KURYR_NEUTRON_DEFAULT_ROUTER="router1"
      KURYR_SG_DRIVER="policy"
      KURYR_SUBNET_DRIVER="namespace"
-      
+
      LOGFILE="/opt/stack/logs/devstacklog.txt"
      LOG_COLOR="False"
      ML2_L3_PLUGIN="ovn-router,trunk,qos"
      OCTAVIA_AMP_IMAGE_FILE="/tmp/test-only-amphora-x64-haproxy-ubuntu-bionic.qcow2"
      OCTAVIA_AMP_IMAGE_NAME="test-only-amphora-x64-haproxy-ubuntu-bionic"
      OCTAVIA_AMP_IMAGE_SIZE="3"
-      
+
      OVN_BRANCH="v20.06.2"
      OVN_BUILD_FROM_SOURCE="True"
      OVN_DBS_LOG_LEVEL="dbg"
      OVN_L3_CREATE_PUBLIC_NETWORK="True"
-      
+
      VAR_RUN_PATH="/usr/local/var/run"
-      
+
      RABBIT_PASSWORD="secretrabbit"
      RECLONE="no"
      SERVICE_PASSWORD="secretservice"
      SERVICE_TOKEN="password"
-      
+
      LIBS_FROM_GIT=cinder,devstack,devstack-gate,devstack-plugin-container,glance,keystone,kuryr-kubernetes,kuryr-tempest-plugin,neutron,nova,octavia,octavia-tempest-plugin,ovn-octavia-provider,placement,python-octaviaclient,requirements,swift,tempest
-      
+
      TEMPEST_PLUGINS="/opt/stack/kuryr-tempest-plugin /opt/stack/octavia-tempest-plugin"
-      
+
      # enabled plugins
      enable_plugin devstack-plugin-container https://opendev.org/openstack/devstack-plugin-container
      enable_plugin kuryr-kubernetes https://opendev.org/openstack/kuryr-kubernetes
@@ -166,7 +166,7 @@ write_files:
      enable_plugin octavia https://opendev.org/openstack/octavia
      enable_plugin octavia-tempest-plugin https://opendev.org/openstack/octavia-tempest-plugin
      enable_plugin ovn-octavia-provider https://opendev.org/openstack/ovn-octavia-provider
-      
+
      [[post-config|$OCTAVIA_CONF]]
      [api_settings]
      enabled_provider_drivers = amphora:'Octavia Amphora driver',ovn:'Octavia OVN driver'
@@ -174,7 +174,7 @@ runcmd:
  - [apt, purge, '-y', python3-pyasn1-modules]
  - [apt, purge, '-y', python3-simplejson]
  - [su, -, ubuntu, -c, "git clone https://github.com/gryf/vmstrap"]
-  - [su, -, ubuntu, -c, "vmstrap/bootstrap.sh"]
+  - [su, -, ubuntu, -c, "vmstrap/bootstrap.sh -c"]
  - [rm, -fr, /home/ubuntu/vmstrap]
  - [su, -, ubuntu, -c, "echo 'export HOST_IP=192.168.10.10' >> .bashrc"]
  - [su, -, ubuntu, -c, "cp /tmp/local.conf /home/ubuntu/devstack/"]
@@ -186,3 +186,4 @@ boxpy_data:
  disk_size: 50GB
  advanced:
    nic2: intnet
+  version: 20.04
--- a/examples/kuryr_multinode_worker.yaml
+++ b/examples/kuryr_multinode_worker.yaml
@@ -34,28 +34,28 @@ write_files:
    content: |
      [[local|localrc]]
      disable_all_services
-      
+
      # Cinder
      disable_service c-bak
      disable_service c-vol
-      
+
      # Dstat (started to fail recently)
      disable_service dstat
-      
+
      # Horizon
      disable_service horizon
-      
+
      # kubernetes
      disable_service kubernetes-master
      enable_service kubernetes-worker
-      
+
      # kuryr
      enable_service kuryr-daemon
      disable_service kuryr-kubernetes
-      
+
      # Nova
      enable_service n-cpu
-      
+
      # Neutron
      enable_service neutron
      enable_service ovn-controller
@@ -63,27 +63,27 @@ write_files:
      enable_service ovn-octavia-provider
      enable_service ovs-vswitchd
      enable_service ovsdb-server
-      
+
      # Placement API
      enable_service placement-client
-      
+
      # Neutron services cd
      enable_service q-ovn-metadata-agent
      disable_service q-svc
-      
+
      # tempest
      disable_service tempest
-      
+
      # tls
      disable_service tls-proxy
-      
+
      # Vars
      ADMIN_PASSWORD="secretadmin"
      DATABASE_HOST="192.168.10.10"
      DATABASE_PASSWORD="secretdatabase"
      ENABLE_CHASSIS_AS_GW="False"
      GLANCE_HOSTPORT="192.168.10.10:9292"
-     
+
      # turn on ovn-provider
      KURYR_ENFORCE_SG_RULES="False"
      KURYR_EP_DRIVER_OCTAVIA_PROVIDER="ovn"
@@ -91,18 +91,18 @@ write_files:
      KURYR_LB_ALGORITHM="SOURCE_IP_PORT"
      KURYR_NEUTRON_DEFAULT_ROUTER="router1"
      VAR_RUN_PATH="/usr/local/var/run"
-      
+
      KURYR_ENABLED_HANDLERS="vif,endpoints,service,namespace,pod_label,policy,kuryrnetworkpolicy,kuryrnetwork,kuryrport,kuryrloadbalancer"
      KURYR_SG_DRIVER="policy"
      KURYR_SUBNET_DRIVER="namespace"
-      
-      
+
+
      OVN_BRANCH="v20.06.2"
      OVN_BUILD_FROM_SOURCE="True"
      OVN_DBS_LOG_LEVEL="dbg"
      OVN_L3_CREATE_PUBLIC_NETWORK="True"
-      
-      
+
+
      LIBVIRT_TYPE="qemu"
      LOGFILE="/opt/stack/logs/devstacklog.txt"
      LOG_COLOR="False"
@@ -113,9 +113,9 @@ write_files:
      SERVICE_HOST="192.168.10.10"
      SERVICE_PASSWORD="secretservice"
      SERVICE_TOKEN="password"
-      
+
      LIBS_FROM_GIT=cinder,devstack,devstack-gate,devstack-plugin-container,glance,keystone,kuryr-kubernetes,kuryr-tempest-plugin,neutron,nova,octavia,placement,python-octaviaclient,requirements,swift,tempest
-      
+
      # enabled plugins
      enable_plugin devstack-plugin-container https://opendev.org/openstack/devstack-plugin-container
      enable_plugin kuryr-kubernetes https://opendev.org/openstack/kuryr
@@ -123,7 +123,7 @@ runcmd:
  - [apt, purge, '-y', python3-pyasn1-modules]
  - [apt, purge, '-y', python3-simplejson]
  - [su, -, ubuntu, -c, "git clone https://github.com/gryf/vmstrap"]
-  - [su, -, ubuntu, -c, "vmstrap/bootstrap.sh"]
+  - [su, -, ubuntu, -c, "vmstrap/bootstrap.sh -c"]
  - [rm, -fr, /home/ubuntu/vmstrap]
  - [su, -, ubuntu, -c, "echo 'export HOST_IP=192.168.10.11' >> .bashrc"]
  - [su, -, ubuntu, -c, "cp /tmp/local.conf /home/ubuntu/devstack/"]
@@ -135,3 +135,4 @@ boxpy_data:
  disk_size: 50GB
  advanced:
    nic2: intnet
+  version: 20.04
--- a/examples/wmaker.yaml
+++ b/examples/wmaker.yaml
@@ -1,5 +1,6 @@
 packages:
  - build-essential
+  - exuberant-ctags
  - gettext
  - libfontconfig1-dev
  - libgif-dev
@@ -21,12 +22,16 @@ packages:
  - libxrender-dev
  - libxt-dev
  - make
+  - mc
  - sharutils
+  - silversearcher-ag
+  - tmux
+  - vim-nox
  - xinit
 runcmd:
  - [su, -, ubuntu, -c, "git clone https://github.com/gryf/wmaker -b experimental"]
  - [su, -, ubuntu, -c, "git clone https://github.com/gryf/vmstrap"]
-  - [su, -, ubuntu, -c, "vmstrap/bootstrap.sh"]
+  - [su, -, ubuntu, -c, "vmstrap/bootstrap.sh -c"]
  - [rm, -fr, /home/ubuntu/vmstrap]
 boxpy_data:
  key: vm
Author	SHA1	Message	Date
gryf	b2457d497e	Add experimental Debian support. Note, there is something weird with those images, as the will segfault on the first run (I've checked that with latest Debian 11 images, maybe it will change in the future), so there is forced reboot after certain amount of time as a workaround.	2023-04-18 21:48:19 +02:00
gryf	15a6ecb540	Use predefined os types for vbox VMs. It might have impact on how operating system inside VM is treated by VirtualBox, so let's stick with proper distro for every OS. As an outcome, it turns out, that for graphical side of the vm, for Linux OS the vmsvga driver is recommended, as a little more amount of vram.	2023-04-18 21:39:56 +02:00
gryf	10543cb506	Don't create vm till it's still there during rebuild	2023-04-02 10:05:06 +02:00
gryf	1f1084f294	Readme update.	2023-02-26 19:20:11 +01:00
gryf	f5ed3d37ac	readme update	2023-02-22 21:10:59 +01:00
gryf	1483f49461	Added debug info about ssh connection to the vm	2023-02-22 21:08:43 +01:00
gryf	f0282874f8	Added information regarding url for the dl image	2023-02-12 11:45:50 +01:00
gryf	4db0b422b8	Added missing fname from Centos class constructor	2023-02-12 11:45:19 +01:00
gryf	c3ee529d95	Make 22.04 default version for ubuntu.	2022-12-24 12:29:49 +01:00