CAELinux 2011 EC2 Image: Monday, November 28, 2011 How to Migrate a local Ubuntu Virtual Machine to EC2 J. Cugnoni 2011 (or in other words how I have prepared the CAELinux 2011 EC2 image...) Procedure based on http://www.ioncannon.net/system-administration/1246/converting-from-virtualbox-or-vmware-to-ec2-now-easier-than-ever/ 1) install Ubuntu (or CAELinux) in VirtualBox a) use entire disk, single root partition b) in the VM, install openssh, libuuid-perl c) in the VM, install ec2 kernel : sudo apt-get install linux-image-ec2 d) n the VM, create grub menu: sudo nano /boot/grub/menu.lst content: default 0 timeout 1 title UBEC2 root (hd0,0) kernel /boot/vmlinuz-2.6.32-340-ec2 root=/dev/sda1 initrd /boot/initrd.img-2.6.32-340-ec2 e) update fstab: sudo nano /etc/fstab, change uuid to /dev/sda1 for root f) update initramfs: sudo update-initramfs -u -k all g) copy /boot to /boot/boot: sudo cp -Rp /boot/ /boot/boot h) customize image to boot from ec2: - remove hostname : sudo rm -f /etc/hostname - copy the following to /etc/ec2-get-credentials.sh (default admin user is ‘caelinux’…): ------ #!/bin/bash # # ec2-get-credentials - Retrieve the ssh credentials and add to authorized_keys # # Based on /usr/local/sbin/ec2-get-credentials from Amazon's ami-20b65349 # # modified by J.Cugnoni for CAELinux (admin user is caelinux) adminprofile=/home/caelinux prog=$(basename $0) logger="logger -t $prog" public_key_url=http://169.254.169.254/1.0/meta-data/public-keys/0/openssh-key public_key_file=/tmp/openssh_id.pub public_key_ephemeral=/mnt/openssh_id.pub authorized_keys=$adminprofile/.ssh/authorized_keys # Wait for the network to come up. perl -MIO::Socket::INET -e ' until(new IO::Socket::INET("169.254.169.254:80")){print"Waiting for network...n";sleep 1} ' | $logger # Try to get the ssh public key from instance data. curl --silent --fail -o $public_key_file $public_key_url test -d $adminprofile/.ssh || mkdir -p -m 700 $adminprofile/.ssh if [ $? -eq 0 -a -e $public_key_file ] ; then if ! grep -s -q -f $public_key_file $authorized_keys then cat $public_key_file >> $authorized_keys $logger "New ssh key added to $authorized_keys from $public_key_url" fi chmod 600 $authorized_keys rm -f $public_key_file # Try to get the ssh public key from ephemeral storage elif [ -e $public_key_ephemeral ] ; then if ! grep -s -q -f $public_key_ephemeral $authorized_keys then cat $public_key_ephemeral >> $authorized_keys $logger "New ssh key added to $authorized_keys from $public_key_ephemeral" fi chmod 600 $authorized_keys chmod 600 $public_key_ephemeral fi ----- then make the script executable: sudo chmod a+x /etc/ec2-get-credentials.sh and install curl: sudo apt-get install curl setup to launch the script on startup: sudo nano /etc/rc.local add /etc/ec2-get-credentials.sh as follows: #!/bin/sh -e # # rc.local # # This script is executed at the end of each multiuser runlevel. # Make sure that the script will "exit 0" on success or any other # value on error. # # In order to enable or disable this script just change the execution # bits. # # By default this script does nothing. # if ssh host keys are not ok, regenerate them if [[ ! -f /etc/ssh/ssh_host_rsa_key ]]; then dpkg-reconfigure openssh-server fi # ec2 get ssh credentials /etc/ec2-get-credentials.sh # make sure that local users can connect to local X server id 0 if [ -z `grep $HOSTNAME /etc/X0.hosts` ]; then echo "localhost" > /etc/X0.hosts echo $HOSTNAME >> /etc/X0.hosts fi exit 0 i) customize image for remote desktop: add the following to /home/caelinux/.profile and /etc/skel/.profile: xhost +$HOSTNAME +localhost export LIBGL_ALWAYS_INDIRECT=1 and install neatx-server sudo add-apt-repository ppa:freenx-team/ppa sudo apt-get update sudo apt-get install neatx-server j) remove predefined passwords (note this will break the local virtual machine, so you can do it later in EC2 and rebundling the image of a running instance): sudo nano /etc/shadow and replace the hash key by ‘x’, example: caelinux:$6$vn.zGgeu$PCresA6UMVEdT/pSgOFjkZi1IOr5wYFzUU63EHwtn7oWLiiz9MolKxL51w8RTdLYVxQ6iJx0DmBv6dN3Oq6Fp.:15305:0:99999:7::: becomes: caelinux:x:15305:0:99999:7::: 2) convert disk image to raw and compress: VBoxManage internalcommands converttoraw myosimage.vdi myosimage.img gzip myosimage.img 3) package in EC2: a) run a “dummy” instance (use m1.large to have local storage) b) create a new volume of the same size as the disk image, attach the volume to instance as /dev/sdf c) copy the compressed image to instance local storage: on instance: sudo mkdir /mnt/image; sudo chmod a+rwx /mnt/image on local pc: sudo scp -i ec2key.pem myosimage.img.gz user@instance-ip:/mnt/image on instance: gzip -d /mnt/image/myosimage.img.gz d) replicate image to volume: sudo dd if=myosimage.gz of=/dev/sdf e) create a snapshot of the volume => get snapshot id f) find the right pvgrub AKI on EC2: see http://docs.amazonwebservices.com/AWSEC2/latest/UserGuide/index.html?UserProvidedkernels.html we need a hd00 type of PVGRUB AKI g) register image using commande line (does not work well with Web interface): caelinux@caelinux:~$ ec2-register -n "AMI-name" -d "description" --kernel aki-60695814 -a x86_64 --root-device-name /dev/sda1 -b /dev/sda=snap-483eaa20:25:true -b /dev/sdb=ephemeral0 4) test and enjoy!! Finally here are some additions, specific to CAELinux 2011: - Install NX server (Neatx here) # add caelinux2010ec2 packages to caelinux 2011 sudo add-apt-repository ppa:freenx-team/ppa sudo apt-get update sudo apt-get install neatx-server - IMPORTANT NOTE: it is a bad idea to install cloud-init as it breaks the boot sequence for an unknown reason (I tried to find out but no answer yet) - todo: remove password and setup and check SSH key based login, check ephemeral, if not rebundle from command line