Add experimental Debian support.

Note, there is something weird with those images, as the will segfault
on the first run (I've checked that with latest Debian 11 images, maybe
it will change in the future), so there is forced reboot after certain
amount of time as a workaround.

README.rst

@@ -2,8 +2,8 @@
 box.py
 ======
 
-Box.py is a simple automation tool meant to run Ubuntu, Fedora or Centos Stream
-cloud images on top of VirtualBox.
+Box.py is a simple automation tool meant to run Ubuntu, Fedora, Centos Stream
+or Debian cloud images on top of VirtualBox.
 
 What it does is simply download official cloud image, set up VM, tweak it up
 and do the initial pre-configuration using generated config drive.
@@ -16,7 +16,7 @@ weird named options for ``vboxmanage`` ;P)
 Requirements
 ------------
 
-- Python >=3.7
+- Python >=3.8
 
   - `pyyaml`_
   - `requests`_
@@ -24,7 +24,7 @@ Requirements
 - Virtualbox (obviously)
 - ``mkisofs`` or ``genisoimage`` command for generating ISO image
 - ``wget`` command for fetching images
-- ``sha256sum`` command for checksum check
+- ``sha256sum`` and ``sha512sum`` commands for checksum check
 - ``qemu-img`` from *qemu-utils* package command for converting between images
   formats
 
@@ -39,7 +39,7 @@ your operating system, or by using virtualenv for Python requirements, i.e.:
 
    $ python -m virtualenv .venv
    $ . .venv/bin/activate
-   (.venv) $ pip install requirements.txt
+   (.venv) $ pip install -r requirements.txt
 
 then you can issue:
 
@@ -55,7 +55,7 @@ or simply link it somewhere in the path:
    $ chmod +x ~/bin/boxpy
 
 and now you can issue some command. For example, to spin up a VM with Ubuntu
-18.04 with one CPU, 2GB of memory and 10GB of disk:
+18.04 with one CPU, 1GB of memory and 6GB of disk:
 
 .. code:: shell-session
 
@@ -91,17 +91,23 @@ use it ad-hoc, or place on your ``.bashrc`` or whatever:
 
 Currently, following commands are available:
 
-- ``list`` - for quickly listing all/running VMs
-- ``info`` - to get summary about VM
-- ``destroy`` - that is probably obvious one
+- ``completion`` - as described above
 - ``create`` - create new VM
+- ``destroy`` - that is probably obvious one
+- ``info`` - to get summary about VM
+- ``list`` - for quickly listing all/running VMs
 - ``rebuild`` - recreate specified VM
 - ``ssh`` - connect to the VM using ssh
-- ``completion`` - as described above
+- ``start`` - stop the running VM
+- ``stop`` - start stopped VM
 
 All of the commands have a range of options, and can be examined by using
 ``--help`` option.
 
+
+YAML Configuration
+------------------
+
 What is more interesting though, is the fact, that you can pass your own
 `cloud-init`_ yaml file, so that VM can be provisioned in easy way.
 
@@ -203,6 +209,19 @@ configuration additional NIC for virtual machine, i.e:
      advanced:
        nic2: intnet
 
+To select image from local file system, it is possible to set one by providing
+it under ``boxpy_data.image`` key:
+
+.. code:: yaml
+
+   …
+   boxpy_data:
+     image: /path/to/the/qcow2/image
+     default_user: cloud-user
+
+Note, that default_user is also needed to be provided, as there is no guess,
+what is the default username for cloud-init configured within provided image.
+
 
 License
 -------

box.py

@@ -23,9 +23,10 @@ __version__ = "1.3"
 CACHE_DIR = os.environ.get('XDG_CACHE_HOME', os.path.expanduser('~/.cache'))
 CLOUD_IMAGE = "ci.iso"
 FEDORA_RELEASE_MAP = {'32': '1.6', '33': '1.2', '34': '1.2'}
+DEBIAN_CODENAME_MAP = {'12': 'bookworm', '11': 'bullseye', '10': 'buster'}
 TYPE_MAP = {'HardDisk': 'disk', 'DVD': 'dvd', 'Floppy': 'floppy'}
 DISTRO_MAP = {'ubuntu': 'Ubuntu', 'fedora': 'Fedora',
-              'centos': 'Centos Stream'}
+              'centos': 'Centos Stream', 'debian': 'Debian'}
 META_DATA_TPL = string.Template('''\
 instance-id: $instance_id
 local-hostname: $vmhostname
@@ -46,9 +47,9 @@ ssh:
   emit_keys_to_console: false
 boxpy_data:
   cpus: 1
-  disk_size: 10240
+  disk_size: 6144
   key: ~/.ssh/id_rsa
-  memory: 2048
+  memory: 1024
 '''
 COMPLETIONS = {'bash': '''\
 _boxpy() {
@@ -138,8 +139,9 @@ _boxpy() {
             fi
             ;;
         create|rebuild)
-            items=(--cpus --disable-nested --disk-size --distro --forwarding
-                --key --memory --hostname --port --config --version --type)
+            items=(--cpus --disable-nested --disk-size --default-user --distro
+                --forwarding --image --key --memory --hostname --port --config
+                --version --type)
             if [[ ${prev} == ${cmd} ]]; then
                 if [[ ${cmd} = "rebuild" ]]; then
                     _vms_comp vms
@@ -159,8 +161,8 @@ _boxpy() {
                         _ssh_identityfile
                         ;;
                     --distro)
-                        COMPREPLY=( $(compgen -W "ubuntu fedora centos" \
-                                -- ${cur}) )
+                        COMPREPLY=( $(compgen -W "ubuntu fedora centos
+                            debian" -- ${cur}) )
                         ;;
                     --type)
                         COMPREPLY=( $(compgen -W "gui headless sdl separate" \
@@ -267,6 +269,10 @@ class BoxVBoxFailure(BoxError):
     pass
 
 
+class BoxConfError(BoxError):
+    pass
+
+
 class FakeLogger:
     """
     print based "logger" class. I like to use 'end' parameter of print
@@ -358,18 +364,20 @@ class FakeLogger:
 
 class Config:
     ATTRS = ('cpus', 'config', 'creator', 'disable_nested', 'disk_size',
-             'distro', 'forwarding', 'hostname', 'key', 'memory', 'name',
-             'port', 'version', 'username')
+             'distro', 'default_user', 'forwarding', 'hostname', 'image',
+             'key', 'memory', 'name', 'port', 'version', 'username')
 
     def __init__(self, args, vbox=None):
         self.advanced = None
         self.distro = None
+        self.default_user = None
         self.cpus = None
         self.creator = None
         self.disable_nested = 'False'
         self.disk_size = None
         self.forwarding = {}
         self.hostname = None
+        self.image = None
         self.key = None
         self.memory = None
         self.name = args.name  # this one is not stored anywhere
@@ -412,6 +420,10 @@ class Config:
                 continue
             setattr(self, attr, str(val))
 
+        # sort out case, where there is image/default-user provided
+        if self.image:
+            self._update_distros_with_custom_image()
+
         # set distribution and version if not specified by user
         if not self.distro:
             self.distro = 'ubuntu'
@@ -498,7 +510,7 @@ class Config:
             self.ssh_key_path = os.path.join(os.path.expanduser("~/.ssh"),
                                              self.ssh_key_path)
         if not os.path.exists(self.ssh_key_path):
-            raise BoxNotFound(f'Cannot find ssh public key: {self.key}')
+            raise BoxConfError(f'Cannot find ssh public key: {self.key}')
 
     def _set_defaults(self):
         conf = yaml.safe_load(USER_DATA)
@@ -553,6 +565,18 @@ class Config:
 
         self._conf = conf
 
+    def _update_distros_with_custom_image(self):
+        self.image = os.path.abspath(self.image)
+        self.distro = 'custom'
+        if not self.username:
+            self.username = self.default_user
+        DISTROS['custom'] = {'username': self.default_user,
+                             'realname': 'custom os',
+                             'img_class': CustomImage,
+                             'amd64': 'x86_64',
+                             'image': self.image,
+                             'default_version': '0'}
+
     def _update(self, source, update):
         for key, val in update.items():
             if isinstance(val, collections.abc.Mapping):
@@ -562,6 +586,17 @@ class Config:
         return source
 
 
+class OsTypes:
+    def ubuntu(conf):
+        return "Ubuntu%s_64" % conf.version.replace('.', '_')
+
+    def fedora(conf):
+        return "Fedora_64"
+
+    def debian(conf):
+        return "Debian%s_64" % conf.version
+
+
 class VBoxManage:
     """
     Class for dealing with vboxmanage commands
@@ -720,6 +755,12 @@ class VBoxManage:
             LOG.fatal('Failed to create VM:\n%s', out.stderr)
             return None
 
+        if out.stdout.startswith('WARNING:'):
+            LOG.fatal('Created crippled VM:\n%s\nFix the issue with '
+                      'VirtualBox, remove the dead VM and start over.',
+                      out.stdout)
+            return None
+
         for line in out.stdout.split('\n'):
             if line.startswith('UUID:'):
                 self.uuid = line.split('UUID:')[1].strip()
@@ -736,7 +777,15 @@ class VBoxManage:
                '--acpi', 'on',
                '--audio', 'none',
                '--nic1', 'nat',
-               '--natpf1', f'boxpyssh,tcp,,{port},,22']
+               '--natpf1', f'boxpyssh,tcp,,{port},,22',
+               '--graphicscontroller', 'vmsvga',
+               '--vram', '16']
+
+        if hasattr(OsTypes, conf.distro):
+            cmd.extend(['--ostype', getattr(OsTypes, conf.distro)(conf)])
+        else:
+            cmd.extend(['--ostype', 'Linux_64'])
+
         for count, (hostport, vmport) in enumerate(conf.forwarding.items(),
                                                    start=1):
             cmd.extend(['--natpf1', f'custom-pf-{count},tcp,,{hostport},'
@@ -885,11 +934,12 @@ class VBoxManage:
 class Image:
     URL = ""
     IMG = ""
+    CHECKSUMTOOL = 'sha256sum'
 
-    def __init__(self, vbox, version, arch, release):
+    def __init__(self, vbox, version, arch, release, fname=None):
         self.vbox = vbox
         self._tmp = tempfile.mkdtemp(prefix='boxpy_')
-        self._img_fname = None
+        self._img_fname = fname
 
     def convert_to_vdi(self, disk_img, size):
         LOG.info('Converting and resizing "%s", new size: %s', disk_img, size)
@@ -935,7 +985,7 @@ class Image:
             return False
 
         if os.path.exists(os.path.join(CACHE_DIR, self._img_fname)):
-            cmd = ['sha256sum', os.path.join(CACHE_DIR, self._img_fname)]
+            cmd = [self.CHECKSUMTOOL, os.path.join(CACHE_DIR, self._img_fname)]
             calulated_sum = Run(cmd).stdout.split(' ')[0]
             LOG.details('Checksum for image: %s, expected: %s', calulated_sum,
                         expected_sum)
@@ -949,7 +999,8 @@ class Image:
             return True
 
         fname = os.path.join(CACHE_DIR, self._img_fname)
-        LOG.header('Downloading image %s', self._img_fname)
+        LOG.header('Downloading image %s from %s', self._img_fname,
+                   self._img_url)
         Run(['wget', '-q', self._img_url, '-O', fname])
 
         if not self._checksum():
@@ -968,7 +1019,7 @@ class Ubuntu(Image):
     URL = "https://cloud-images.ubuntu.com/releases/%s/release/%s"
     IMG = "ubuntu-%s-server-cloudimg-%s.img"
 
-    def __init__(self, vbox, version, arch, release):
+    def __init__(self, vbox, version, arch, release, fname=None):
         super().__init__(vbox, version, arch, release)
         self._img_fname = self.IMG % (version, arch)
         self._img_url = self.URL % (version, self._img_fname)
@@ -987,13 +1038,37 @@ class Ubuntu(Image):
         return expected_sum
 
 
+class Debian(Image):
+    URL = "https://cloud.debian.org/images/cloud/%s/daily/latest/%s"
+    IMG = "debian-%s-generic-%s-daily.qcow2"
+    CHECKSUMTOOL = 'sha512sum'
+
+    def __init__(self, vbox, version, arch, release, fname=None):
+        super().__init__(vbox, version, arch, release)
+        self._img_fname = self.IMG % (version, arch)
+        self._img_url = self.URL % (release, self._img_fname)
+        self._checksum_file = 'SHA512SUMS'
+        self._checksum_url = self.URL % (release, self._checksum_file)
+
+    def _get_checksum(self, fname):
+        expected_sum = None
+        Run(['wget', self._checksum_url, '-q', '-O', fname])
+        with open(fname) as fobj:
+            for line in fobj.readlines():
+                if self._img_fname in line:
+                    expected_sum = line.split(' ')[0]
+                    break
+
+        return expected_sum
+
+
 class Fedora(Image):
     URL = ("https://download.fedoraproject.org/pub/fedora/linux/releases/%s/"
            "Cloud/%s/images/%s")
     IMG = "Fedora-Cloud-Base-%s-%s.%s.qcow2"
     CHKS = "Fedora-Cloud-%s-%s-%s-CHECKSUM"
 
-    def __init__(self, vbox, version, arch, release):
+    def __init__(self, vbox, version, arch, release, fname=None):
         super().__init__(vbox, version, arch, release)
         self._img_fname = self.IMG % (version, release, arch)
         self._img_url = self.URL % (version, arch, self._img_fname)
@@ -1016,10 +1091,10 @@ class Fedora(Image):
 
 class CentosStream(Image):
     URL = "https://cloud.centos.org/centos/%s-stream/%s/images/%s"
-    IMG = '.*(CentOS-Stream-GenericCloud-%s-[0-9]+\.[0-9].%s.qcow2).*'
+    IMG = '.*(CentOS-Stream-GenericCloud-%s-[0-9]+.[0-9].%s.qcow2).*'
     CHKS = "CHECKSUM"
 
-    def __init__(self, vbox, version, arch, release):
+    def __init__(self, vbox, version, arch, release, fname=None):
         super().__init__(vbox, version, arch, release)
         self._checksum_file = '%s-centos-stream-%s-%s' % (self.CHKS, version,
                                                           arch)
@@ -1064,11 +1139,18 @@ class CentosStream(Image):
         return expected_sum
 
 
+class CustomImage(Image):
+
+    def _download_image(self):
+        # just use provided image
+        return True
+
+
 DISTROS = {'ubuntu': {'username': 'ubuntu',
                       'realname': 'ubuntu',
                       'img_class': Ubuntu,
                       'amd64': 'amd64',
-                      'default_version': '20.04'},
+                      'default_version': '22.04'},
            'fedora': {'username': 'fedora',
                       'realname': 'fedora',
                       'img_class': Fedora,
@@ -1078,15 +1160,22 @@ DISTROS = {'ubuntu': {'username': 'ubuntu',
                       'realname': 'centos',
                       'img_class': CentosStream,
                       'amd64': 'x86_64',
-                      'default_version': '8'}}
+                      'default_version': '8'},
+           'debian': {'username': 'debian',
+                      'realname': 'debian',
+                      'img_class': Debian,
+                      'amd64': 'amd64',
+                      'default_version': '11'}}
 
 
 def get_image_object(vbox, version, image='ubuntu', arch='amd64'):
     release = None
     if image == 'fedora':
         release = FEDORA_RELEASE_MAP[version]
+    if image == 'debian':
+        release = DEBIAN_CODENAME_MAP[version]
     return DISTROS[image]['img_class'](vbox, version, DISTROS[image]['amd64'],
-                                       release)
+                                       release, DISTROS[image].get('image'))
 
 
 class IsoImage:
@@ -1137,7 +1226,8 @@ def vmcreate(args, conf=None):
     if not conf:
         try:
             conf = Config(args)
-        except BoxNotFound:
+        except BoxConfError as err:
+            LOG.fatal(f'Configuration error: {err.args[0]}.')
             return 7
         except yaml.YAMLError:
             LOG.fatal(f'Cannot read or parse file `{args.config}` as YAML '
@@ -1161,7 +1251,9 @@ def vmcreate(args, conf=None):
     if not vbox.create_controller('SATA', 'sata'):
         return 4
 
-    for key in ('distro', 'hostname', 'key', 'version'):
+    for key in ('distro', 'hostname', 'key', 'version', 'image', 'username'):
+        if getattr(conf, key) is None:
+            continue
         if not vbox.setextradata(key, getattr(conf, key)):
             return 5
 
@@ -1207,6 +1299,7 @@ def vmcreate(args, conf=None):
            f'ssh://{username}@localhost:{vbox.vm_info["port"]}',
            'sudo cloud-init status']
     try:
+        counter = 0
         while True:
             out = Run(cmd)
             LOG.debug('Out: %s', out.stdout)
@@ -1226,6 +1319,15 @@ def vmcreate(args, conf=None):
                                           f'VM with ssh for user {username}. '
                                           f'Check output in debug mode.')
                 time.sleep(3)
+                counter += 1
+                if counter == 8 and conf.distro == 'debian':
+                    counter += 1
+                    # there is something odd with debian cloud images, as they
+                    # segfault on first run. after ~20 seconds there should
+                    # already be panic, reset machine should help
+                    vbox.poweroff()
+                    time.sleep(3)
+                    vbox.poweron(args.type)
                 continue
 
             LOG.info(' done')
@@ -1405,6 +1507,13 @@ def vmrebuild(args):
         conf.disk_size = vbox.get_media_size(disk_path)
 
     vmdestroy(args)
+
+    # Wait till VM is gone
+    while True:
+        vbox = VBoxManage(args.name)
+        if not vbox.get_vm_info():
+            break
+
     vmcreate(args, conf)
     return 0
 
@@ -1430,11 +1539,14 @@ def connect(args):
         return 16
 
     username = conf.username or DISTROS[conf.distro]["username"]
-    return Run(['ssh', '-o', 'StrictHostKeyChecking=no',
-                '-o', 'UserKnownHostsFile=/dev/null',
-                '-i', conf.ssh_key_path[:-4],
-                f'ssh://{username}'
-                f'@localhost:{conf.port}'], False).returncode
+    cmd = ['ssh', '-o', 'StrictHostKeyChecking=no',
+           '-o', 'UserKnownHostsFile=/dev/null',
+           '-i', conf.ssh_key_path[:-4],
+           f'ssh://{username}'
+           f'@localhost:{conf.port}']
+    LOG.debug('Connecting to vm `%s` using command:\n%s', args.name,
+              ' '.join(cmd))
+    return Run(cmd, False).returncode
 
 
 def _set_vmstate(name, state):
@@ -1493,10 +1605,16 @@ def main():
                         help="Alternative user-data template filepath")
     create.add_argument('-d', '--distro', help="Image name. 'ubuntu' is "
                         "default")
+    create.add_argument('-e', '--default-user', help="Default cloud-init user "
+                        "to be used with custom image (--image param). "
+                        "Without image it will make no effect.")
     create.add_argument('-f', '--forwarding', action='append', help="expose "
                         "port from VM to the host. It should be in format "
                         "'hostport:vmport'. this option can be used multiple "
                         "times for multiple ports.")
+    create.add_argument('-i', '--image', help="custom qcow2 image filepath. "
+                        "Note, that it requires to provide --default-user as "
+                        "well.")
     create.add_argument('-k', '--key', help="SSH key to be add to the config "
                         "drive. Default ~/.ssh/id_rsa")
     create.add_argument('-m', '--memory', help="amount of memory in "
@@ -1538,10 +1656,16 @@ def main():
     rebuild.add_argument('-c', '--config',
                          help="Alternative user-data template filepath")
     rebuild.add_argument('-d', '--distro', help="Image name.")
+    rebuild.add_argument('-e', '--default-user', help="Default cloud-init "
+                         "user to be used with custom image (--image param). "
+                         "Without image it will make no effect.")
     rebuild.add_argument('-f', '--forwarding', action='append', help="expose "
                          "port from VM to the host. It should be in format "
                          "'hostport:vmport'. this option can be used multiple "
                          "times for multiple ports.")
+    rebuild.add_argument('-i', '--image', help="custom qcow2 image filepath. "
+                         "Note, that it requires to provide --default-user as "
+                         "well.")
     rebuild.add_argument('-k', '--key',
                          help='SSH key to be add to the config drive')
     rebuild.add_argument('-m', '--memory', help='amount of memory in '
@@ -1584,6 +1708,10 @@ def main():
 
     args = parser.parse_args()
 
+    if 'image' in args and 'default_user' not in args:
+        parser.error('Parameter --image requires --default-user')
+        return 22
+
     LOG.set_verbose(args.verbose, args.quiet)
 
     if 'func' not in args and args.version:

examples/devstack-ubuntu-20.04.yaml

@@ -29,7 +29,7 @@ runcmd:
   - [apt, purge, '-y', python3-pyasn1-modules]
   - [apt, purge, '-y', python3-simplejson]
   - [su, -, ubuntu, -c, "git clone https://github.com/gryf/vmstrap"]
-  - [su, -, ubuntu, -c, "vmstrap/bootstrap.sh"]
+  - [su, -, ubuntu, -c, "vmstrap/bootstrap.sh -c"]
   - [rm, -fr, /home/ubuntu/vmstrap]
   - [su, -, ubuntu, -c, "cp /tmp/local.conf /home/ubuntu/devstack/"]
   - [su, -, ubuntu, -c, "echo 'export HOST_IP=10.0.2.15' >> .bashrc"]

examples/kuryr_multinode_master.yaml

@@ -174,7 +174,7 @@ runcmd:
   - [apt, purge, '-y', python3-pyasn1-modules]
   - [apt, purge, '-y', python3-simplejson]
   - [su, -, ubuntu, -c, "git clone https://github.com/gryf/vmstrap"]
-  - [su, -, ubuntu, -c, "vmstrap/bootstrap.sh"]
+  - [su, -, ubuntu, -c, "vmstrap/bootstrap.sh -c"]
   - [rm, -fr, /home/ubuntu/vmstrap]
   - [su, -, ubuntu, -c, "echo 'export HOST_IP=192.168.10.10' >> .bashrc"]
   - [su, -, ubuntu, -c, "cp /tmp/local.conf /home/ubuntu/devstack/"]
@@ -186,3 +186,4 @@ boxpy_data:
   disk_size: 50GB
   advanced:
     nic2: intnet
+  version: 20.04

examples/kuryr_multinode_worker.yaml

@@ -123,7 +123,7 @@ runcmd:
   - [apt, purge, '-y', python3-pyasn1-modules]
   - [apt, purge, '-y', python3-simplejson]
   - [su, -, ubuntu, -c, "git clone https://github.com/gryf/vmstrap"]
-  - [su, -, ubuntu, -c, "vmstrap/bootstrap.sh"]
+  - [su, -, ubuntu, -c, "vmstrap/bootstrap.sh -c"]
   - [rm, -fr, /home/ubuntu/vmstrap]
   - [su, -, ubuntu, -c, "echo 'export HOST_IP=192.168.10.11' >> .bashrc"]
   - [su, -, ubuntu, -c, "cp /tmp/local.conf /home/ubuntu/devstack/"]
@@ -135,3 +135,4 @@ boxpy_data:
   disk_size: 50GB
   advanced:
     nic2: intnet
+  version: 20.04

examples/wmaker.yaml

@@ -1,5 +1,6 @@
 packages:
   - build-essential
+  - exuberant-ctags
   - gettext
   - libfontconfig1-dev
   - libgif-dev
@@ -21,12 +22,16 @@ packages:
   - libxrender-dev
   - libxt-dev
   - make
+  - mc
   - sharutils
+  - silversearcher-ag
+  - tmux
+  - vim-nox
   - xinit
 runcmd:
   - [su, -, ubuntu, -c, "git clone https://github.com/gryf/wmaker -b experimental"]
   - [su, -, ubuntu, -c, "git clone https://github.com/gryf/vmstrap"]
-  - [su, -, ubuntu, -c, "vmstrap/bootstrap.sh"]
+  - [su, -, ubuntu, -c, "vmstrap/bootstrap.sh -c"]
   - [rm, -fr, /home/ubuntu/vmstrap]
 boxpy_data:
   key: vm