gryf/openstack
1
0
Fork 0
mirror of https://github.com/gryf/openstack.git synced 2025-12-17 11:30:24 +01:00
Code Issues Projects Releases Wiki Activity

Open-source the OpenStack DB migration scripts and workshop

Browse Source
This commit is contained in:
PTC
2018-11-07 14:11:17 -08:00
parent 002a4a71c7
commit 875239ca08
20 changed files with 1053 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

54
migration-scripts/Code-of-Conduct.md Normal file
View File

@@ -0,0 +1,54 @@
# Oath Open Source Code of Conduct
## Summary
This Code of Conduct is our way to encourage good behavior and discourage bad behavior in our open source community. We invite participation from many people to bring different perspectives to support this project. We pledge to do our part to foster a welcoming and professional environment free of harassment. We expect participants to communicate professionally and thoughtfully during their involvement with this project.
Participants may lose their good standing by engaging in misconduct. For example: insulting, threatening, or conveying unwelcome sexual content. We ask participants who observe conduct issues to report the incident directly to the project's Response Team at opensource-conduct@oath.com. Oath will assign a respondent to address the issue. We may remove harassers from this project.
This code does not replace the terms of service or acceptable use policies of the websites used to support this project. We acknowledge that participants may be subject to additional conduct terms based on their employment which may govern their online expressions.
## Details
This Code of Conduct makes our expectations of participants in this community explicit.
* We forbid harassment and abusive speech within this community.
* We request participants to report misconduct to the project’s Response Team.
* We urge participants to refrain from using discussion forums to play out a fight.
### Expected Behaviors
We expect participants in this community to conduct themselves professionally. Since our primary mode of communication is text on an online forum (e.g. issues, pull requests, comments, emails, or chats) devoid of vocal tone, gestures, or other context that is often vital to understanding, it is important that participants are attentive to their interaction style.
* **Assume positive intent.** We ask community members to assume positive intent on the part of other people’s communications. We may disagree on details, but we expect all suggestions to be supportive of the community goals.
* **Respect participants.** We expect participants will occasionally disagree. Even if we reject an idea, we welcome everyone’s participation. Open Source projects are learning experiences. Ask, explore, challenge, and then respectfully assert if you agree or disagree. If your idea is rejected, be more persuasive not bitter.
* **Welcoming to new members.** New members bring new perspectives. Some may raise questions that have been addressed before. Kindly point them to existing discussions. Everyone is new to every project once.
* **Be kind to beginners.** Beginners use open source projects to get experience. They might not be talented coders yet, and projects should not accept poor quality code. But we were all beginners once, and we need to engage kindly.
* **Consider your impact on others.** Your work will be used by others, and you depend on the work of others. We expect community members to be considerate and establish a balance their self-interest with communal interest.
* **Use words carefully.** We may not understand intent when you say something ironic. Poe’s Law suggests that without an emoticon people will misinterpret sarcasm. We ask community members to communicate plainly.
* **Leave with class.** When you wish to resign from participating in this project for any reason, you are free to fork the code and create a competitive project. Open Source explicitly allows this. Your exit should not be dramatic or bitter.
### Unacceptable Behaviors
Participants remain in good standing when they do not engage in misconduct or harassment. To elaborate:
* **Don't be a bigot.** Calling out project members by their identity or background in a negative or insulting manner. This includes, but is not limited to, slurs or insinuations related to protected or suspect classes e.g. race, color, citizenship, national origin, political belief, religion, sexual orientation, gender identity and expression, age, size, culture, ethnicity, genetic features, language, profession, national minority statue, mental or physical ability.
* **Don't insult.** Insulting remarks about a person’s lifestyle practices.
* **Don't dox.** Revealing private information about other participants without explicit permission.
* **Don't intimidate.** Threats of violence or intimidation of any project member.
* **Don't creep.** Unwanted sexual attention or content unsuited for the subject of this project.
* **Don't disrupt.** Sustained disruptions in a discussion.
* **Let us help.** Refusal to assist the Response Team to resolve an issue in the community.
We do not list all forms of harassment, nor imply some forms of harassment are not worthy of action. Any participant who *feels* harassed or *observes* harassment, should report the incident. Victim of harassment should not address grievances in the public forum, as this often intensifies the problem. Report it, and let us address it off-line.
### Reporting Issues
If you experience or witness misconduct, or have any other concerns about the conduct of members of this project, please report it by contacting our Response Team at opensource-conduct@oath.com who will handle your report with discretion. Your report should include:
* Your preferred contact information. We cannot process anonymous reports.
* Names (real or usernames) of those involved in the incident.
* Your account of what occurred, and if the incident is ongoing. Please provide links to or transcripts of the publicly available records (e.g. a mailing list archive or a public IRC logger), so that we can review it.
* Any additional information that may be helpful to achieve resolution.
After filing a report, a representative will contact you directly to review the incident and ask additional questions. If a member of the Oath Response Team is named in an incident report, that member will be recused from handling your incident. If the complaint originates from a member of the Response Team, it will be addressed by a different member of the Response Team. We will consider reports to be confidential for the purpose of protecting victims of abuse.
### Scope
Oath will assign a Response Team member with admin rights on the project and legal rights on the project copyright. The Response Team is empowered to restrict some privileges to the project as needed. Since this project is governed by an open source license, any participant may fork the code under the terms of the project license. The Response Team’s goal is to preserve the project if possible, and will restrict or remove participation from those who disrupt the project.
This code does not replace the terms of service or acceptable use policies that are provided by the websites used to support this community. Nor does this code apply to communications or actions that take place outside of the context of this community. Many participants in this project are also subject to codes of conduct based on their employment. This code is a social-contract that informs participants of our social expectations. It is not a terms of service or legal contract.
## License and Acknowledgment.
This text is shared under the [CC-BY-4.0 license](https://creativecommons.org/licenses/by/4.0/). This code is based on a study conducted by the [TODO Group](https://todogroup.org/) of many codes used in the open source community. If you have feedback about this code, contact our Response Team at the address listed above.

26
migration-scripts/Contributing.md Normal file
View File

@@ -0,0 +1,26 @@
# How to contribute
First, thanks for taking the time to contribute to our project! The following information provides a guide for making contributions.
## Code of Conduct
By participating in this project, you agree to abide by the [Oath Code of Conduct](Code-of-Conduct.md). Everyone is welcome to submit a pull request or open an issue to improve the documentation, add improvements, or report bugs.
## How to Ask a Question
If you simply have a question that needs an answer, [create an issue](https://help.github.com/articles/creating-an-issue/), and label it as a question.
## How To Contribute
### Report a Bug or Request a Feature
If you encounter any bugs while using this software, or want to request a new feature or enhancement, feel free to [create an issue](https://help.github.com/articles/creating-an-issue/) to report it, make sure you add a label to indicate what type of issue it is.
### Contribute Code
Pull requests are welcome for bug fixes. If you want to implement something new, please [request a feature first](#report-a-bug-or-request-a-feature) so we can discuss it.
#### Creating a Pull Request
Before you submit any code, we need you to agree to our [Contributor License Agreement](https://yahoocla.herokuapp.com/); this ensures we can continue to protect your contributions under an open source license well into the future.
Please follow [best practices](https://github.com/trein/dev-best-practices/wiki/Git-Commit-Best-Practices) for creating git commits.
When your code is ready to be submitted, you can [submit a pull request](https://help.github.com/articles/creating-a-pull-request/) to begin the code review process.

9
migration-scripts/LICENSE Normal file
View File

@@ -0,0 +1,9 @@
MIT License
Copyright 2018, Oath Inc.
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

43
migration-scripts/README.md Normal file
View File

@@ -0,0 +1,43 @@
# Migration-Scripts
> Scripts and tools to migrate to the OpenStack Ocata release.
## Table of Contents
- [Background](#background)
- [Install](#install)
- [Usage](#usage)
- [Contribute](#contribute)
- [License](#license)
## Background
We needed to update Oath's OpenStack deployment from Juno to the Ocata release, and we realized there weren't any good resources for doing this. So, we wrote our own. If you're using OpenStack Juno and need to upgrade to Ocata (this can also be adapted for other releases), these scripts might be what you need.
## Install
No installation is necessary, the main script will install Python and all other necessary packages for the migration.
## Usage
Available scripts include:
* clone.sh - Used to clone all repositories that are required for the DB migration.
* cp_build.sh - Copies build.tar.gz to all db hosts.
* do_migration.sh - Migrates hosts to Ocata.
* remove_deleted_instances.py - Remove instances that are deleted in the process.
Each of these scripts can be run with the -h flag for more details.
Here's an example of the DB migration command:
```
./do_migration.sh <RW_HOSTNAME> 3306 root `sudo keydbgetkey mysqlroot` "rabbit://ostk_rabbit_user:<PASSWORD>@<MQ1HOST>:5672/ostk_rabbit_vhost" mysql+pymysql://nova:<NOVA_PASSWORD>@<RW_HOSTNAME>:3306
```
## Contribute
Please refer to [the contributing.md file](Contributing.md) for information about how to get involved. We welcome issues, questions, and pull requests. Pull Requests are welcome.
## License
This project is licensed under the terms of the MIT open source license. Please refer to [LICENSE](LICENSE) for the full terms.

63
migration-scripts/migration-workshop/README.md Normal file
View File

@@ -0,0 +1,63 @@
Database Migration Workshop Instructions
```bash
# Install libvirt and its plugin for vagrant.
# You may need to do something different but this
# is a way that works on CentOS 7.
sudo yum install -y qemu libvirt libvirt-devel ruby-devel gcc qemu-kvm dkms make qt libgomp patch kernel-headers kernel-devel binutils glibc-headers glibc-devel font-forge
sudo yum install -y https://releases.hashicorp.com/vagrant/1.9.6/vagrant_1.9.6_x86_64.rpm
sudo vagrant plugin install vagrant-libvirt
# Create and ssh to the Vagrant VM
sudo vagrant up
sudo vagrant ssh
# Become root user
sudo su
# Deploy Juno and boot a VM
/vagrant/juno.sh
# Manually check to see the VM is there
source kolla/openrc
source venv/bin/activate
nova list
ps aux | grep qemu
# Deactivate the venv, this is important for later steps
deactivate
# Shut down all Openstack services except DB
/vagrant/rm_juno_services.sh
# Clone down the DB migration scripts
git clone https://github.com/yahoo/openstack-collab.git
cd openstack-collab/migration-scripts/scripts/
# Create a venv for each version of each component to be migrated
./clone.sh
tar xf build.tar.gz
# Run the actual DB migration
/vagrant/run_migration.sh
# Take a copy of the now Ocata DB
/vagrant/take_db_dump.sh
# Destroy all old docker data, deploy a fresh Ocata cluster
/vagrant/ocata/ocata.sh
# Actually we only need the DB right now, so stop everything but that.
/vagrant/ocata/stop_ocata_services.sh
# Put the migrated DB dump into MariaDB
/vagrant/ocata/restore_dump.sh
# Redeploy with kolla-ansible again so service users and endpoints are re-made.
/vagrant/ocata/redeploy.sh
# Get the openstack client
cd /home/vagrant
virtualenv ocata-venv
source ocata-venv/bin/activate
pip install python-openstackclient
source admin-openrc.sh
# As you can see the old server you booted is still there
openstack server list
# However it's not actually running if you check ps.
ps aux | grep qemu
# This can simply be fixed with a stop/start.
openstack server stop demo1
openstack server start demo1
ps aux | grep qemu
# Let's boot up another server to make sure that works
openstack server create --flavor m1.tiny --image cirros --key-name demo-keypair --nic net-id=$(openstack network list -f value -c ID) demo2
openstack server list
ps aux | grep qemu
```

13
migration-scripts/migration-workshop/Vagrantfile vendored Normal file
View File

@@ -0,0 +1,13 @@
# -*- mode: ruby -*-
# vi: set ft=ruby :
Vagrant.configure("2") do |config|
config.ssh.forward_agent = true
config.vm.box = "centos/7"
config.vm.network "private_network", ip: "192.168.33.10"
config.vm.provider "libvirt" do |v|
v.memory = 8192
v.cpus = 4
v.storage :file, :size => '40G'
end
end

205
migration-scripts/migration-workshop/genenv Executable file
View File

@@ -0,0 +1,205 @@
#!/bin/bash
#
# This script generates a minimal set of environment variables to allow
# the openstack containers to operate. It is creating a configuration
# suitable for an all-in-one installation of openstack.
#
# It also creates a suitable 'openrc' for use with the installed system.
# Move to top level directory
REAL_PATH=$(python -c "import os,sys;print os.path.realpath('$0')")
cd "$(dirname "$REAL_PATH")/.."
MY_IP=192.168.33.10
MY_DEV=eth1
echo MY_IP=$MY_IP
echo MY_DEV=$MY_DEV
# Admin user password
ADMIN_USER_PASSWORD=steakfordinner
# Database
HOST_IP=$MY_IP
MARIADB_ROOT_PASSWORD=kolla
PASSWORD=12345
# Host
ADMIN_TENANT_NAME=admin
PUBLIC_IP=$HOST_IP
# Logging
DEBUG_LOGGING=false
VERBOSE_LOGGING=true
NOVA_LOG_DIR=
NEUTRON_LOG_DIR=/var/log/neutron
NOVA_API_LOG_FILE=
NOVA_CONDUCTOR_LOG_FILE=
NOVA_SCHEDULER_LOG_FILE=
NOVA_COMPUTE_LOG_FILE=
NOVA_CONSOLEAUTH_LOG_FILE=
NOVA_NOVNCPROXY_LOG_FILE=
NEUTRON_SERVER_LOG_FILE=
NEUTRON_L3_AGENT_LOG_FILE=
NEUTRON_LINUXBRIDGE_AGENT_LOG_FILE=
NEUTRON_METADATA_AGENT_LOG_FILE=
# RabbitMQ
RABBITMQ_SERVICE_HOST=$HOST_IP
RABBIT_USER=guest
RABBIT_PASSWORD=guest
# Networking Options are nova or neutron
NETWORK_MANAGER=neutron
# Keystone
KEYSTONE_ADMIN_TOKEN=$PASSWORD
KEYSTONE_DB_PASSWORD=kolla
KEYSTONE_ADMIN_PASSWORD=$PASSWORD
KEYSTONE_PUBLIC_SERVICE_HOST=$HOST_IP
KEYSTONE_ADMIN_SERVICE_HOST=$HOST_IP
KEYSTONE_AUTH_PROTOCOL=http
# Glance
GLANCE_DB_NAME=glance
GLANCE_DB_USER=glance
GLANCE_DB_PASSWORD=kolla
GLANCE_KEYSTONE_USER=glance
GLANCE_KEYSTONE_PASSWORD=glance
GLANCE_API_SERVICE_HOST=$HOST_IP
GLANCE_REGISTRY_SERVICE_HOST=$HOST_IP
# Nova
NOVA_DB_PASSWORD=nova
NOVA_DB_NAME=nova
NOVA_DB_USER=nova
NOVA_KEYSTONE_USER=nova
NOVA_KEYSTONE_PASSWORD=nova
NOVA_API_SERVICE_HOST=$HOST_IP
NOVA_METADATA_API_SERVICE_HOST=$HOST_IP
NOVA_EC2_SERVICE_HOST=$HOST_IP
NOVA_PUBLIC_INTERFACE=$MY_DEV
NOVA_FLAT_INTERFACE=eth1
ENABLED_APIS=ec2,osapi_compute,metadata
METADATA_HOST=$HOST_IP
NOVA_NOVNC_PROXY_PORT=6080
NOVA_NOVNC_PROXY_SERVICE_HOST=0.0.0.0
NOVA_NOVNC_BASE_ADDRESS=${PUBLIC_IP}
NOVA_NOVNC_PROXYCLIENT_IP=${PUBLIC_IP}
# Neutron
NEUTRON_DB_NAME=neutron
NEUTRON_DB_USER=neutron
NEUTRON_DB_PASSWORD=password
NEUTRON_KEYSTONE_USER=neutron
NEUTRON_KEYSTONE_PASSWORD=neutron
NEUTRON_SERVER_SERVICE_HOST=$HOST_IP
NEUTRON_API_PASTE_CONFIG=/usr/share/neutron/api-paste.ini
# Neutron ML2 Plugin
TYPE_DRIVERS=flat,vxlan
TENANT_NETWORK_TYPES=flat,vxlan
MECHANISM_DRIVERS=linuxbridge,l2population
# Neutron Linux Bridge Agent
NEUTRON_FLAT_NETWORK_NAME=physnet1
NEUTRON_FLAT_NETWORK_INTERFACE=eth1
# Nova/Neutron
NEUTRON_SHARED_SECRET=sharedsecret
# Heat
HEAT_DB_NAME=heat
HEAT_DB_PASSWORD=kolla
HEAT_KEYSTONE_PASSWORD=heat
HEAT_API_SERVICE_HOST=$HOST_IP
HEAT_API_CFN_SERVICE_HOST=$HOST_IP
HEAT_API_CFN_URL_HOST=$HOST_IP
cat > ./openrc <<EOF
export OS_AUTH_URL="http://${KEYSTONE_PUBLIC_SERVICE_HOST}:5000/v2.0"
export OS_USERNAME=$ADMIN_TENANT_NAME
export OS_PASSWORD=$ADMIN_USER_PASSWORD
export OS_TENANT_NAME=$ADMIN_TENANT_NAME
EOF
cat > ./compose/openstack.env <<EOF
DEBUG_LOGGING=$DEBUG_LOGGING
VERBOSE_LOGGING=$VERBOSE_LOGGING
NOVA_LOG_DIR=$NOVA_LOG_DIR
NEUTRON_LOG_DIR=$NEUTRON_LOG_DIR
NOVA_API_LOG_FILE=$NOVA_API_LOG_FILE
NOVA_CONDUCTOR_LOG_FILE=$NOVA_CONDUCTOR_LOG_FILE
NOVA_SCHEDULER_LOG_FILE=$NOVA_SCHEDULER_LOG_FILE
NOVA_COMPUTE_LOG_FILE=$NOVA_COMPUTE_LOG_FILE
NOVA_CONSOLEAUTH_LOG_FILE=$NOVA_CONSOLEAUTH_LOG_FILE
NOVA_NOVNCPROXY_LOG_FILE=$NOVA_NOVNCPROXY_LOG_FILE
NEUTRON_SERVER_LOG_FILE=$NEUTRON_SERVER_LOG_FILE
NEUTRON_L3_AGENT_LOG_FILE=$NEUTRON_L3_AGENT_LOG_FILE
NEUTRON_LINUXBRIDGE_AGENT_LOG_FILE=$NEUTRON_LINUXBRIDGE_AGENT_LOG_FILE
NEUTRON_METADATA_AGENT_LOG_FILE=$NEUTRON_METADATA_AGENT_LOG_FILE
TYPE_DRIVERS=$TYPE_DRIVERS
TENANT_NETWORK_TYPES=$TENANT_NETWORK_TYPES
MECHANISM_DRIVERS=$MECHANISM_DRIVERS
ADMIN_USER_PASSWORD=$ADMIN_USER_PASSWORD
ADMIN_TENANT_NAME=$ADMIN_TENANT_NAME
DB_ROOT_PASSWORD=$MARIADB_ROOT_PASSWORD
FLAT_INTERFACE=$NOVA_FLAT_INTERFACE
GLANCE_API_SERVICE_HOST=$GLANCE_API_SERVICE_HOST
GLANCE_DB_NAME=$GLANCE_DB_NAME
GLANCE_DB_PASSWORD=$GLANCE_DB_PASSWORD
GLANCE_DB_USER=$GLANCE_DB_USER
GLANCE_KEYSTONE_PASSWORD=$GLANCE_KEYSTONE_PASSWORD
GLANCE_KEYSTONE_USER=$GLANCE_KEYSTONE_USER
GLANCE_REGISTRY_SERVICE_HOST=$GLANCE_REGISTRY_SERVICE_HOST
KEYSTONE_ADMIN_PASSWORD=$KEYSTONE_ADMIN_PASSWORD
KEYSTONE_ADMIN_SERVICE_HOST=$KEYSTONE_ADMIN_SERVICE_HOST
KEYSTONE_ADMIN_SERVICE_PORT=35357
KEYSTONE_ADMIN_TOKEN=$KEYSTONE_ADMIN_TOKEN
KEYSTONE_AUTH_PROTOCOL=$KEYSTONE_AUTH_PROTOCOL
KEYSTONE_DB_PASSWORD=$KEYSTONE_DB_PASSWORD
KEYSTONE_PUBLIC_SERVICE_HOST=$KEYSTONE_PUBLIC_SERVICE_HOST
MARIADB_SERVICE_HOST=$HOST_IP
MARIADB_ROOT_PASSWORD=$MARIADB_ROOT_PASSWORD
NETWORK_MANAGER=$NETWORK_MANAGER
NOVA_API_SERVICE_HOST=$NOVA_API_SERVICE_HOST
NOVA_METADATA_API_SERVICE_HOST=$NOVA_METADATA_API_SERVICE_HOST
ENABLED_APIS=$ENABLED_APIS
METADATA_HOST=$METADATA_HOST
NOVA_DB_NAME=$NOVA_DB_NAME
NOVA_DB_PASSWORD=$NOVA_DB_PASSWORD
NOVA_DB_USER=$NOVA_DB_USER
NOVA_EC2_API_SERVICE_HOST=$NOVA_EC2_SERVICE_HOST
NOVA_EC2_SERVICE_HOST=$NOVA_EC2_SERVICE_HOST
NOVA_KEYSTONE_PASSWORD=$NOVA_KEYSTONE_PASSWORD
NOVA_KEYSTONE_USER=$NOVA_KEYSTONE_USER
NOVA_NOVNC_PROXY_SERVICE_HOST=$NOVA_NOVNC_PROXY_SERVICE_HOST
NOVA_NOVNC_PROXY_PORT=$NOVA_NOVNC_PROXY_PORT
NOVA_NOVNC_BASE_ADDRESS=${NOVA_NOVNC_BASE_ADDRESS}
NOVA_NOVNC_PROXYCLIENT_IP=${NOVA_NOVNC_PROXYCLIENT_IP}
NEUTRON_SHARED_SECRET=$NEUTRON_SHARED_SECRET
NEUTRON_DB_NAME=$NEUTRON_DB_NAME
NEUTRON_DB_USER=$NEUTRON_DB_USER
NEUTRON_DB_PASSWORD=$NEUTRON_DB_PASSWORD
NEUTRON_KEYSTONE_USER=$NEUTRON_KEYSTONE_USER
NEUTRON_KEYSTONE_PASSWORD=$NEUTRON_KEYSTONE_PASSWORD
NEUTRON_SERVER_SERVICE_HOST=$NEUTRON_SERVER_SERVICE_HOST
NEUTRON_API_PASTE_CONFIG=$NEUTRON_API_PASTE_CONFIG
NEUTRON_FLAT_NETWORK_NAME=$NEUTRON_FLAT_NETWORK_NAME
NEUTRON_FLAT_NETWORK_INTERFACE=$NEUTRON_FLAT_NETWORK_INTERFACE
HEAT_DB_NAME=$HEAT_DB_NAME
HEAT_DB_PASSWORD=$HEAT_DB_PASSWORD
HEAT_KEYSTONE_PASSWORD=$HEAT_KEYSTONE_PASSWORD
HEAT_API_SERVICE_HOST=$HEAT_API_SERVICE_HOST
PUBLIC_INTERFACE=$NOVA_PUBLIC_INTERFACE
PUBLIC_IP=$HOST_IP
PUBLIC_IP=$PUBLIC_IP
RABBITMQ_PASS=$RABBIT_PASSWORD
RABBITMQ_SERVICE_HOST=$RABBITMQ_SERVICE_HOST
RABBITMQ_USER=$RABBIT_USER
RABBIT_PASSWORD=$RABBIT_PASSWORD
RABBIT_USERID=$RABBIT_USER
HEAT_API_CFN_SERVICE_HOST=$HEAT_API_CFN_SERVICE_HOST
EOF
echo Please customize your FLAT_INTERFACE to a different network then your
echo main network. The FLAT_INTERFACE is used for inter-VM communication.
echo the FLAT_INTERFACE should not have an IP address assigned.

56
migration-scripts/migration-workshop/juno.sh Executable file
View File

@@ -0,0 +1,56 @@
#!/bin/bash
# Copyright 2018, Oath Inc.
# Licensed under the terms of the MIT license. See LICENSE file for terms.
set -Eexo pipefail
setenforce permissive
yum install -y epel-release
yum install -y python-pip python-devel libffi-devel gcc openssl-devel libselinux-python git curl mysql
pip install virtualenv
[ -e venv ] || virtualenv venv
source venv/bin/activate
curl -L https://github.com/docker/compose/releases/download/1.8.1/docker-compose-$(uname -s)-$(uname -m) -o /usr/bin/docker-compose
chmod +x /usr/bin/docker-compose
curl -o/etc/yum.repos.d/docker-ce.repo https://download.docker.com/linux/centos/docker-ce.repo
yum -y install docker-ce
systemctl start docker
curl https://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt?h=kilo-eol | grep -v '^aioeventlet' | grep -v '^cryptography=' | grep -v '^pyOpenSSL=' | grep -v '^cffi=' | grep -v 'SQLAlchemy' | grep -v 'MySQL-python' | grep -v 'PyMySQL' | grep -v 'alembic' > up_cons_file
echo "aioeventlet===0.5.2" >> up_cons_file
grep -v 'oslo.i18n' up_cons_file | grep -v 'oslo.db' | grep -v '^kombu' > up_cons_file-BAK
echo "oslo.i18n===1.7.0" >> up_cons_file-BAK
echo "oslo.db===1.7.0" >> up_cons_file-BAK
echo "kombu===3.0.30" >> up_cons_file-BAK
mv up_cons_file-BAK up_cons_file
pip install python-keystoneclient==1.6.0 python-glanceclient==0.19.0 python-novaclient==2.26.0 python-heatclient==0.6.0 python-neutronclient==2.6.0 -c up_cons_file
rm up_cons_file
[ -e kolla ] || git clone https://github.com/openstack/kolla.git
cd kolla
git checkout tags/juno-eol
cp /vagrant/genenv ./tools/genenv
./tools/genenv
cd compose
for f in *.yml ; do grep -v 'name:' $f > tmp.yml && mv tmp.yml $f ; done
cd ..
export COMPOSE_API_VERSION=1.18
echo 'NOVA_CONSOLEAUTH_LOG_FILE=/var/log/nova/nova-consoleauth.log' >> compose/openstack.env
echo 'NOVA_NOVNCPROXY_LOG_FILE=/var/log/nova/nova-vncproxy.log' >> compose/openstack.env
./tools/kolla start
source openrc
MY_IP=192.168.33.10
mysql -h${MY_IP} -uroot -pkolla -e 'DROP DATABASE heat;'
docker stop compose_heatapi_1 compose_heatengine_1 compose_horizon_1
./tools/init-runonce
[ -e ~/.ssh/id_rsa ] || ssh-keygen -t rsa -f ~/.ssh/id_rsa
nova keypair-add --pub-key ~/.ssh/id_rsa.pub demo-keypair
neutron subnet-update demo-subnet --enable_dhcp false
nova boot --flavor m1.tiny --image cirros --key-name demo-keypair --nic net-id=$(neutron net-list -f value -F id) demo1

4
migration-scripts/migration-workshop/ocata/ansible.cfg Normal file
View File

@@ -0,0 +1,4 @@
[defaults]
host_key_checking=False
pipelining=True
forks=100

24
migration-scripts/migration-workshop/ocata/extra.yaml Normal file
View File

@@ -0,0 +1,24 @@
kolla_base_distro: "centos"
kolla_install_type: "source"
openstack_release: "ocata"
kolla_internal_vip_address: "192.168.33.25"
network_interface: "eth1"
enable_heat: "no"
enable_horizon: "no"
database_password: "kolla"
keystone_admin_password: "steakfordinner"
nova_database_password: "nova"
nova_api_database_password: "nova"
rabbitmq_user: "guest"
rabbitmq_password: "guest"
keystone_token_provider: "fernet"
glance_database_password: "kolla"
glance_keystone_password: "glance"
keystone_database_password: "kolla"
nova_keystone_password: "nova"
neutron_database_password: "password"
neutron_keystone_password: "neutron"
placement_keystone_user: "nova"
placement_keystone_password: "nova"

39
migration-scripts/migration-workshop/ocata/ocata.sh Executable file
View File

@@ -0,0 +1,39 @@
#!/bin/bash
# Copyright 2018, Oath Inc.
# Licensed under the terms of the MIT license. See LICENSE file for terms.
set -e
set -x
ROOT_DIR=/vagrant/ocata
yum remove -y docker-ce || :
rm -f /etc/yum.repos.d/docker-ce.repo || :
umount /var/lib/docker/devicemapper/ || :
rm -rf /var/lib/docker/ || :
yum install -y epel-release
yum install -y python-pip python-devel libffi-devel gcc openssl-devel libselinux-python git
pip install -U pip
[ -e ${ROOT_DIR}/kolla-ansible ] || git clone https://github.com/openstack/kolla-ansible.git ${ROOT_DIR}/kolla-ansible -b stable/ocata
pip install ${ROOT_DIR}/kolla-ansible
pip install "ansible>=2,<2.4" virtualenv
[ -e /etc/ansible ] || mkdir /etc/ansible
cp ${ROOT_DIR}/ansible.cfg /etc/ansible/
[ -e /etc/kolla ] || cp -r ${ROOT_DIR}/kolla-ansible/etc/kolla /etc/
kolla-genpwd
mkdir -p /etc/kolla/config/nova/
cat > /etc/kolla/config/nova/nova-compute.conf <<EOF
[libvirt]
virt_type = qemu
cpu_mode = none
EOF
cp ${ROOT_DIR}/kolla-ansible/ansible/inventory/all-in-one ${ROOT_DIR}
kolla-ansible -i ${ROOT_DIR}/all-in-one --extra @${ROOT_DIR}/extra.yaml bootstrap-servers
kolla-ansible -i ${ROOT_DIR}/all-in-one --extra @${ROOT_DIR}/extra.yaml prechecks
kolla-ansible -i ${ROOT_DIR}/all-in-one --extra @${ROOT_DIR}/extra.yaml pull
kolla-ansible -i ${ROOT_DIR}/all-in-one --extra @${ROOT_DIR}/extra.yaml deploy
docker exec openvswitch_db ovs-vsctl set interface eth1 type=internal
kolla-ansible --extra @${ROOT_DIR}/extra.yaml post-deploy
cp /etc/kolla/admin-openrc.sh /home/vagrant

11
migration-scripts/migration-workshop/ocata/redeploy.sh Executable file
View File

@@ -0,0 +1,11 @@
#!/bin/bash
# Copyright 2018, Oath Inc.
# Licensed under the terms of the MIT license. See LICENSE file for terms.
set -e
set -x
ROOT_DIR=/vagrant/ocata
docker rm -f $(docker ps -aq)
kolla-ansible -i ${ROOT_DIR}/all-in-one --extra @${ROOT_DIR}/extra.yaml bootstrap-servers
kolla-ansible -i ${ROOT_DIR}/all-in-one --extra @${ROOT_DIR}/extra.yaml deploy

10
migration-scripts/migration-workshop/ocata/restore_dump.sh Executable file
View File

@@ -0,0 +1,10 @@
#!/bin/bash
# Copyright 2018, Oath Inc.
# Licensed under the terms of the MIT license. See LICENSE file for terms.
mysql -h192.168.33.10 -uroot -pkolla -e "show databases" | grep -v Database | grep -v mysql| grep -v information_schema| gawk '{print "drop database " $1 ";select sleep(0.1);"}' | mysql -h192.168.33.10 -uroot -pkolla
mysql -h192.168.33.10 -uroot -pkolla < ocata_db_dump.sql
cp -r instances/ /var/lib/docker/volumes/nova_compute/_data
cp -r images/ /var/lib/docker/volumes/glance/_data
chown --reference=/var/lib/docker/volumes/nova_compute/_data -R /var/lib/docker/volumes/nova_compute/_data
chown --reference=/var/lib/docker/volumes/glance/_data -R /var/lib/docker/volumes/glance/_data

5
migration-scripts/migration-workshop/ocata/stop_ocata_services.sh Executable file
View File

@@ -0,0 +1,5 @@
#!/bin/bash
# Copyright 2018, Oath Inc.
# Licensed under the terms of the MIT license. See LICENSE file for terms.
docker stop $(docker ps --format '{{.ID}} {{.Names}}' | grep -v maria | awk '{print $1}')

8
migration-scripts/migration-workshop/rm_juno_services.sh Executable file
View File

@@ -0,0 +1,8 @@
#!/bin/bash
# Copyright 2018, Oath Inc.
# Licensed under the terms of the MIT license. See LICENSE file for terms.
source /home/vagrant/venv/bin/activate
source /home/vagrant/kolla/openrc
for SERVICE_USER in keystone heat glance neutron nova; do keystone user-delete $SERVICE_USER; done
docker stop $(docker ps --format '{{.ID}} {{.Names}}' | grep -v maria | awk '{print $1}')

12
migration-scripts/migration-workshop/run_migration.sh Executable file
View File

@@ -0,0 +1,12 @@
#!/bin/bash
# Copyright 2018, Oath Inc.
# Licensed under the terms of the MIT license. See LICENSE file for terms.
MY_IP=192.168.33.10
mysql -h${MY_IP} -uroot -pkolla -e 'UPDATE nova.compute_nodes SET host_ip="192.168.33.10";'
./do_migration.sh ${MY_IP} 3306 root kolla rabbit://guest:guest@${MY_IP}:5672 mysql+pymysql://nova:nova@${MY_IP}:3306
mysql -h${MY_IP} -uroot -pkolla -e 'UPDATE nova_api.cell_mappings SET database_connection="mysql+pymysql://nova:nova@192.168.33.25:3306/nova" WHERE id=2;'
# Only use this if on a very small disk
#mysql -h${MY_IP} -uroot -pkolla -e 'UPDATE nova.compute_nodes SET disk_available_least=free_disk_gb;';
for DIR in $(find /var/lib/docker/volumes/ -type d -name images); do if [ $(ls $DIR/.. | wc -l) -eq "1" ] ; then cp -r $DIR . ; fi ; done
cp -r $(find /var/lib/docker/volumes/ -type d -name instances) .

7
migration-scripts/migration-workshop/take_db_dump.sh Executable file
View File

@@ -0,0 +1,7 @@
#!/bin/bash
# Copyright 2018, Oath Inc.
# Licensed under the terms of the MIT license. See LICENSE file for terms.
MY_IP=192.168.33.10
mysqldump -u root --password=kolla -h $MY_IP --all-databases --result-file=ocata_db_dump.sql
docker stop $(docker ps -q)

188
migration-scripts/scripts/clone.sh Executable file
View File

@@ -0,0 +1,188 @@
#!/usr/bin/env bash
# Copyright 2018, Oath Inc.
# Licensed under the terms of the MIT license. See LICENSE file for terms.
set -Eexo pipefail
# Usage:
# ------
# ./clone.sh [option]
#
# option:
# -h, --help: show brief help
# -b, --baremetal: Include the repositories needed for baremetal
#
# Requirements:
# -------------
# 1. Must have access to github.com
# 2. Must be on RHEL6 or RHEL7
# Purpose/Background:
# -------------------
# This script is used to clone down all repositories that are required for the
# DB migration from juno to ocata. It will also build all of the required
# virtualenvs. The output of this script is a tarball called 'build.tar.gz'.
# This script takes around 30-40 (60 for --baremetal) minutes to run. It is
# useful to have this done and the build.tar.gz file available, prior to
# the actual start of a DB migration.
#
# After this script is finished:
# - Copy this tarball to the DB host of the target migration cluster
# before starting the migration (via e.g. the 'cp_build.sh' script).
# - Unzip the tarball with 'tar -xzf build.tar.gz'. Unzipping the
# tarball will create a 'build' directory.
# - Run the migration script 'do_migration.sh' from within the directory
# that holds the build directory.
usage() {
echo "Usage:"
echo "$0 [option]"
echo " option:"
echo " -h, --help: Display this help message"
echo " -b, --baremetal: Include repository for baremetal (ironic)"
}
if ! full_release=$(cat /etc/redhat-release) ; then
echo "This script must be run on a RHEL machine"
exit 1
fi
set -e
baremetal=0
while (( "$#" )); do
case "$1" in
-b|--baremetal)
baremetal=1
shift
;;
-h|--help)
usage
exit 0
;;
-*|--*=) # unsupported flags
echo "Error: Unsupported flag '$1'" >&2
exit 1
;;
*) # unsupported positional arguments
echo "Error: Unsupported positional argument '$1'" >&2
exit 1
;;
esac
done
echo "RHEL release from /etc/redhat-release: $full_release"
case $full_release in
*"release 6"*)
sudo yum install -y mysql-devel;;
*"release 7"*)
sudo yum install -y mariadb-devel;;
esac
sudo yum install -y git libxml2-devel libxslt-devel libffi-devel openssl-devel libvirt-devel
# Make sure python interpreter and virtualenv are available
sudo yum install -y python python-pip
export PIP_REQUIRE_VIRTUALENV=false
sudo pip install virtualenv
export PIP_REQUIRE_VIRTUALENV=true
rm -rf ~/.cache/pip
components="keystone nova glance neutron"
releases="kilo liberty mitaka newton ocata"
if [ ! -e build ]; then
mkdir build
fi
(
cd build
build_dir=$(pwd)
for comp in $components ; do
(
[ -e "${comp}" ] || git clone "https://github.com/openstack/${comp}.git" "${comp}"
for release in ${releases} ; do
dir_name="${comp}-${release}"
cd "$comp";
if git branch -a | grep -q stable/${release} ; then
tag="stable/${release}"
else
tag="${release}-eol"
fi
cd ..
echo "Processing $dir_name..."
(
cd "${comp}"
git checkout "${tag}"
)
cp -r "${comp}" "${dir_name}"
(
venv_name="venv-${dir_name}"
virtualenv "${venv_name}"
source ${venv_name}/bin/activate
cd "${dir_name}"
# Before installing we need to make a couple changes to the upper-constraints file
if [ "${release}" == "newton" ] ; then
up_cons_tag='stable/newton'
else
up_cons_tag="$tag"
fi
up_cons_file=$dir_name-upper-constraints.txt
curl "https://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt?h=${up_cons_tag}" | grep -v '^aioeventlet' | grep -v '^cryptography=' | grep -v '^pyOpenSSL=' | grep -v '^cffi=' | grep -v 'SQLAlchemy' | grep -v 'MySQL-python' | grep -v 'PyMySQL' | grep -v 'alembic' > "${up_cons_file}"
echo "aioeventlet===0.5.2" >> "${up_cons_file}"
if [ "${release}" == "kilo" ] ; then
echo "Changing i18n version for kilo release"
grep -v 'oslo.i18n' "${up_cons_file}" | grep -v 'oslo.db' | grep -v '^kombu' > "${up_cons_file}-BAK"
echo "oslo.i18n===1.7.0" >> "${up_cons_file}-BAK"
echo "oslo.db===1.7.0" >> "${up_cons_file}-BAK"
echo "kombu===3.0.30" >> "${up_cons_file}-BAK"
cp "${up_cons_file}-BAK" "${up_cons_file}"
fi
# install package and mysql bindings using modified upper-constraints
pip install . MySQL-python PyMySQL -c "${up_cons_file}"
deactivate
)
done
)
rm -rf "${comp}"
done
if [ "$baremetal" == 1 ]; then
# NOTE(rloo): For the ironic DB migration, we only need the ocata
# version of our modified ironic. Instead of adding a bunch of ifs
# above to handle this case, the relevant bits are copy/pasted,
# because this is a one-off and will not be needed for future
# (post-ocata) migrations.
#
# This adds 'ironic-ocata' and 'venv-ironic-ocata' directories.
comp="ironic"
release="ocata"
git clone "https://github.com/openstack/${comp}.git" "${comp}"
dir_name="${comp}-${release}"
branch="stable/${release}"
echo "Processing $dir_name..."
(
cd "${comp}"
git checkout "${branch}"
)
cp -r "${comp}" "${dir_name}"
(
venv_name="venv-${dir_name}"
virtualenv "${venv_name}"
source ${venv_name}/bin/activate
cd "${dir_name}"
up_cons_file=$dir_name-upper-constraints.txt
cp upper-constraints.txt "${up_cons_file}"
# install package and mysql bindings using upper-constraints
pip install . MySQL-python PyMySQL -c "${up_cons_file}"
deactivate
)
rm -rf "${comp}"
fi
)
echo "Zipping all repos into a single tarball..."
tar -czf build.tar.gz build
echo "Cleaning..."
rm -rf build
echo "Success! You can unpack the tarball with the following command: tar -xzf build.tar.gz"

237
migration-scripts/scripts/do_migration.sh Executable file
View File

@@ -0,0 +1,237 @@
#!/usr/bin/env sh
# Copyright 2018, Oath Inc.
# Licensed under the terms of the MIT license. See LICENSE file for terms.
set -Eexo pipefail
#########################################################
########## RUN THIS SCRIPT IN A SCREEN SESSION ##########
#########################################################
# Requirements:
# 1. This script must be run from an openstack db host
# 2. Must have root privs on the machine
# 3. Must be run alongside a build directory that is generated by clone.sh in
# OpenStack/migration-scripts git repo
# 4. It is highly recommended to run this script in a screen session.
# Usage:
# ./do_migration.sh <db_host> <db_port> <db_host_user> <db_host_password> <transport_url> <nova_connection_url> [option]
#
# ./do_migration.sh <RW_HOSTNAME> 3306 root pw "rabbit://ostk_rabbit_user:<RABBIT_PASSWORD>@<MQ1HOST>:<RABBIT_PORT>/ostk_rabbit_vhost" mysql+pymysql://nova:<NOVA_PASSWORD>@<RW_HOSTNAME>:3306 --baremetal
#
# - <transport_url>:
# - if only using 1 MQ node, then the format for transport URL will be the same as above.
# - if we are using more than one MQ, then the transport URL should look like this:
# "rabbit://ostk_rabbit_user:<RABBIT_PASSWORD>@<MQ1_HOST>:<RABBIT_PORT>,ostk_rabbit_user:<RABBIT_PASSWORD>@<MQ2_HOST>:<RABBIT_PORT>/ostk_rabbit_vhost"
# Notice that rabbit:// comes only once before the list of hosts, and /ostk_rabbit_vhost
# comes only once at the very end.
#
# - <RW_HOSTNAME> is the Brooklyn read/write host for the DB.
# - <RABBIT_PASSWORD> is from nova.conf.
# - <MQ1HOST> is the hostname of the Rabbit MQ 1 node.
# - It can be found in the dashboard. Click on clusters on the left. Find the target
# - cluster and click on it. Expand "Hosts" on the resulting page and find "Queue Hosts".
# - <RABBIT_PORT> is 5671 if you are using SSL, and 5672 if you are not using SSL
# - <NOVA_PASSWORD> is the nova user db password.
# The keyname is db_nova_password.
# - [option]
# -b, --baremetal: also do DB migration for ironic
# -h, --help: show brief help
usage() {
echo "Usage:"
echo "$0 <db_host> <db_port> <db_host_user> <db_host_password> <transport_url> <nova_connection_url> [option]"
echo " option:"
echo " -h, --help: Display this help message"
echo " -b, --baremetal: also do DB migration for ironic"
echo "for more information, see comments in the script file"
}
set -e
set -x
baremetal=0
error() {
echo error_msg="FAIL! $1 directory was not found. Run this script https://git.ouroath.com/OpenStack/ocata-migration-scripts/blob/master/clone.sh from a RHEL jumphost to generate the build directory remotely. Then copy the build directory to this machine."
exit 1
}
PARAMS=""
while (( "$#" )); do
case "$1" in
-b|--baremetal)
baremetal=1
shift
;;
-h|--help)
usage
exit 0
;;
-*|--*=) # unsupported flags
echo "Error: Unsupported flag '$1'" >&2
exit 1
;;
*) # preserve positional arguments
PARAMS="$PARAMS $1"
shift
;;
esac
done
# set positional arguments in their proper place
eval set -- "$PARAMS"
if [ $# -ne 6 ] ; then
usage
exit 1
fi
# Make sure git is installed
sudo yum install -y git mysql
db_host="$1"
db_port="$2"
db_user="$3"
db_pass="$4"
transport_url="$5"
nova_db_url="$6"
# Each release to migrate through (Juno, Ocata]
releases="kilo liberty mitaka newton ocata"
# Which components to migrate
if [ "$baremetal" == 1 ]; then
components="keystone nova glance neutron ironic"
else
components="keystone nova glance neutron"
fi
# Let's make sure we have everything that's required installed
sudo yum install -y python python-pip
sudo pip install virtualenv
sudo yum install -y git libxml2-devel libxslt-devel libffi-devel openssl-devel libvirt-devel
# Back up the db first :-)
sql_dump_file="mysqldump.$db_host.`date +"%s"`"
mysqldump -u $db_user --password=$db_pass -h $db_host --all-databases --result-file=$sql_dump_file
# Some necessary preparations to the DB
mysql -u $db_user --password=$db_pass -h $db_host -e "CREATE DATABASE IF NOT EXISTS nova_api;"
mysql -u $db_user --password=$db_pass -h $db_host -e "USE mysql ; GRANT ALL ON nova_api.* TO 'nova'@'%' with GRANT option; FLUSH PRIVILEGES;"
mysql -u $db_user --password=$db_pass -h $db_host -e "CREATE DATABASE IF NOT EXISTS nova_cell0;"
mysql -u $db_user --password=$db_pass -h $db_host -e "USE mysql ; GRANT ALL ON nova_cell0.* TO 'nova'@'%' with GRANT option; FLUSH PRIVILEGES;"
#mysql -u $db_user --password=$db_pass -h $db_host -e "SET GLOBAL FOREIGN_KEY_CHECKS=0;"
mysql -u $db_user --password=$db_pass -h $db_host -e 'USE glance ; CREATE INDEX ix_images_is_public ON images (is_public);' || true
mysql -u $db_user --password=$db_pass -h $db_host -e 'USE glance ; ALTER TABLE image_properties DROP FOREIGN KEY image_properties_ibfk_1;' || true
# Front load some checks to make sure that the necessary directories are in place
if [ ! -e build ]; then
error "build"
fi
cd build
for comp in $components ; do
for release in $releases ; do
if [ $comp == 'ironic' ] && [ $release != 'ocata' ] ; then
continue
else
dir_name="${comp}-$release"
if [ ! -e $dir_name ]; then
error ${dir_name}
fi
venv_name=venv-$dir_name
if [ ! -e $venv_name ]; then
error ${venv_name}
fi
fi
done
done
# use this script to clean up deleted instances
#if [ ! -e cleaner-venv ]; then
# /opt/python/bin/virtualenv -p /opt/python/bin/python2.7 cleaner-venv
#fi
#source cleaner-venv/bin/activate
#pip install mysql-connector-python-rf
#echo "Removing deleted instances...."
#../remove_deleted_instances.py $db_host $db_user $db_pass
#echo "Done."
#deactivate
for comp in $components ; do
echo "Migrating $comp";
# Component will use this config file to talk to the db
db_conf_file=$comp-db.conf
rm -f $db_conf_file
echo "[database]" >> $db_conf_file
if [ $comp == 'nova' ]; then
echo "connection = $nova_db_url/nova" >> $db_conf_file
else
echo "connection = mysql+pymysql://$db_user:$db_pass@$db_host:$db_port/$comp" >> $db_conf_file
fi
echo "[api_database]" >> $db_conf_file
echo "connection = $nova_db_url/nova_api" >> $db_conf_file
echo "[DEFAULT]" >> $db_conf_file
echo "transport_url = $transport_url" >> $db_conf_file
echo "log_dir = /tmp/logs/$comp" >> $db_conf_file
echo "debug = true" >> $db_conf_file
mkdir -p "/tmp/logs/$comp"
for release in $releases ; do
if [ $comp == 'ironic' ] && [ $release != 'ocata' ] ; then
continue
fi
echo "Migrating to $release";
# A mapping from component -> migration command
declare -A migration_command=(["keystone"]="keystone-manage --config-file $db_conf_file db_sync" \
["glance"]="glance-manage --config-file $db_conf_file db_sync" \
["nova"]="nova-manage --config-file $db_conf_file api_db sync ; nova-manage --config-file $db_conf_file db sync" \
["ironic"]="ironic-dbsync --config-file $db_conf_file upgrade" \
["neutron"]="neutron-db-manage --config-file $db_conf_file upgrade heads" \
["horizon"]=":" )
dir_name="${comp}-$release"
venv_name=venv-$dir_name
source $venv_name/bin/activate
# Create cells for ocata
if [ $release == 'newton' ] && [ $comp == 'nova' ] ; then
nova-manage --verbose --config-file $db_conf_file cell_v2 simple_cell_setup
fi
# One time clean-up of null UUIDs
if [ $release == 'kilo' ] && [ $comp == 'nova' ] ; then
nova-manage --config-file $db_conf_file db null_instance_uuid_scan --delete
fi
# Run migration command in the venv and then exit the venv
eval ${migration_command[$comp]}
# Online data migrations
if [ $release != 'kilo' ] && [ $release != 'liberty' ] && [ $comp == 'nova' ] ; then
nova-manage --config-file $db_conf_file db online_data_migrations
fi
# The great Kilo flavor migration of 1934
if [ $release == 'kilo' ] && [ $comp == 'nova' ] ; then
nova-manage --config-file $db_conf_file db migrate_flavor_data --force
mysql -u $db_user --password=$db_pass -h $db_host -e 'USE nova ; UPDATE compute_nodes SET host=hypervisor_hostname;'
fi
deactivate
done
done
# Remove old endpoints (chef deploy will re-populate these with correct values)
mysql -u $db_user --password=$db_pass -h $db_host -e 'USE keystone; DELETE FROM endpoint;'
mysql -u $db_user --password=$db_pass -h $db_host -e 'USE keystone; DELETE FROM service;'
# Remove invalid role assignments (missing user)
# Change legacy "config_drive" key to what is used upstream "configdrive"
# NOTE(jaypipes): This may take a multiple dozens of seconds on a table with
# tens of thousands of records
if [ "$baremetal" == 1 ]; then
mysql -u $db_user --password=$db_pass -h $db_host -e 'USE ironic; UPDATE nodes SET instance_info = REPLACE(instance_info, "\"config_drive\":", "\"configdrive\":");'
fi

39
migration-scripts/scripts/remove_deleted_instances.py Normal file
View File

@@ -0,0 +1,39 @@
#!/usr/bin/env python
# Copyright 2018, Oath Inc.
# Licensed under the terms of the MIT license. See LICENSE file for terms.
"""
Usage:
./remove_deleted_instances.py <db_host> <db_user> <db_pass>
"""
import mysql.connector
import sys
def tuple_to_dict(cur, tup):
return dict(zip(cur.column_names, tup))
uuids = []
cnx = mysql.connector.connect(host=sys.argv[1], user=sys.argv[2], passwd=sys.argv[3], db="nova")
cur1 = cnx.cursor()
cur = cnx.cursor()
cur.execute("SELECT * FROM `instances` WHERE `deleted_at` IS NOT NULL")
for row in cur.fetchall():
row = tuple_to_dict(cur, row)
uuids.append(row['uuid']);
if len(uuids) <= 1:
sys.exit(0)
cur1.execute("SET FOREIGN_KEY_CHECKS = 0;")
uuids = tuple(str(u) for u in uuids)
cur.execute("DELETE FROM instance_id_mappings where uuid IN {}".format(uuids))
cur.execute("DELETE FROM instance_info_caches where instance_uuid IN {}".format(uuids))
cur.execute("DELETE FROM instance_system_metadata where instance_uuid IN {}".format(uuids))
cur.execute("DELETE FROM security_group_instance_association where instance_uuid IN {}".format(uuids))
cur.execute("DELETE FROM instances where uuid IN {}".format(uuids))
cnx.commit()
cur1.execute("SET FOREIGN_KEY_CHECKS = 1;")
cnx.close()