Bloomberg Clustered Private Cloud distribution
chef-bcpc is a set of Chef cookbooks and Ansible playbooks that build a highly-available OpenStack cloud.
The cloud consists of a variety of head nodes (OpenStack controller services, Ceph Mons/Mgrs, etcd, RabbitMQ), work nodes (hypervisors) and storage nodes (Ceph OSDs).
Each type of head node runs its core services in a highly-available manner and the roles of these nodes can be converged into a smaller set of hosts. In addition, the roles of work nodes and storage nodes can also be converged together.
The following instructions will get chef-bcpc up and running on your local machine for development and testing purposes.
NOTE: It is likely possible to build an environment with 16GB of RAM or less if one is willing to make slight modifications to the virtual topology and/or change some of the build settings and overrides. However, we've opted to spec the minimum requirements slightly more aggressively and target hosts with 32GB RAM or more to provide the best out-of-the-box experience.
virtual/topology
, or build your own.
hardware.yml and
topology.yml are used by default. To view a
list of tested topologies and hardware configurations please see
virtual/README.virtual/vagrantbox.json
. The variable vagrant_box
specifies the
Vagrant box we use to build the virtual environment, and vagrant_box_version
specifies
the version of the Vagrant box. These variables are specified per Ansible inventory group
of hosts, and must have a "default" group as is done in the default vagrantbox.json
.virtual/packer/config/variables.json
and set the variables. Depends on the
virtual machine provider, an example can be found at
variables.json.libvirt.example
or variables.json.virtualbox.example.
This step is essential for building a Packer box that's used as a base box image for building
the virtual environment. The variables bcc_apt_key_url
, bcc_apt_url
and vagrant_cacert
are optional,
while others must be set. The variable kernel_version
specifies the Linux kernel version we'd
like to have for the Packer box. While base_box
, base_box_version
, and base_box_provider
specify an official Vagrant box we'd like to use as a baseline for the Packer box, upon which
we make further modifications. Last but not least, the variable output_packer_box_name
specifies
the name we'd like to use when adding the output Packer box to Vagrant.virtual/packer/config/s3.json
can be set up to leverage a pre-built packer box. An example can be found at
s3.json.libvirt.example
or s3.json.virtualbox.example. Run make target make download-packer-box
and make upload-packer-box
to download/upload a packer box.make create-packer-box
. This will create a Packer box and add it to Vagrant
with the name specified by output_packer_box_name
.virtual/vagrantbox.json
accordingly. When a local custom box built by Packer
is used, the variable vagrant_box
needs to be set to the name of the Packer box (aka, the same as
output_packer_box_name
in virtual/packer/config/variables.json
), and vagrant_box_version
should be set to 0.make create all
would always use the Packer box, unless virtual/vagrantbox.json
is specified otherwise.make destroy
, and then
run make destroy-packer-box
to clean up the Packer box.hardware.overrides.yml
and topology.overrides.yml
, respectively, and make
changes to them instead.bcc_http_proxy_url
and bcc_https_proxy_url
respectively in
virtual/packer/config/variables.json
.Download and install the latest version of Packer
wget https://releases.hashicorp.com/packer/1.6.6/packer_1.6.6_linux_amd64.zip -O ~/packer_1.6.6_linux.zip
sudo apt install unzip
sudo unzip ~/packer_1.6.6_linux.zip -d /usr/local/bin
Create a Python virtual environment (virtualenv) and activate it
python3 -mvenv venv
source venv/bin/activate
pip install 'pip>=19.1.1' wheel
pip install PyYaml ansible netaddr pyOpenSSL 'cryptography>=3.0,<38.0.0'
To create a libvirt build (the default), first install the following packages and plugins:
sudo apt-get install build-essential dnsmasq libguestfs-tools libvirt-dev pkg-config qemu-utils
vagrant plugin install vagrant-libvirt vagrant-mutate
Alternatively, to create a VirtualBox build, install the following plugin and set the following environment variables:
vagrant plugin install vagrant-vbguest
export VAGRANT_DEFAULT_PROVIDER=virtualbox
export VAGRANT_VAGRANTFILE=Vagrantfile.virtualbox
Use the following commands to create a virtual build:
make generate-chef-databags
make create-packer-box
make create all
To clean up the build:
make destroy
make destroy-packer-box
You may also want to change CPU model from qemu64
to kvm64
in
ansible/playbooks/roles/common/defaults/main/chef.yml
chef_environment:
name: virtual
override_attributes:
bcpc:
nova:
cpu_config:
cpu_mode: custom
cpu_model: kvm64
To switch from the default libvirt provider to the virtualbox provider, as far as the build is concerned, you can just remove the mutated libvirt box and then set VAGRANT_DEFAULT_PROVIDER and VAGRANT_VAGRANTFILE environment variables as described above. However since you must also make sure that the different hypervisors don't both try to control the CPU virtualization facilities, it is best to remove the mutated box and then simply reboot your development host.
This would look something like this:
$ rm -rf ~/.vagrant.d/boxes/bento-VAGRANTSLASH-ubuntu-20.04/202206.03.0/libvirt/
$ rm -rf ~/.vagrant.d/boxes/bento-VAGRANTSLASH-ubuntu-22.04/202206.13.0/libvirt/
$ sudo reboot
TBD
Currently, most development is done by a team at Bloomberg L.P. but we would like to build a community around this project. PRs and issues are welcomed. If you are interested in joining the team at Bloomberg L.P. please see available opportunities at the Bloomberg L.P. careers site.
This project is licensed under the Apache 2.0 License - see the LICENSE file for details.
chef-bcpc is built with the following open source software:
Thanks to all of these communities for producing this software!