DP Code Development VM
Getting started developing on the DP code has always been tricky because of its dependencies (notably phpBB) and complexity to set up. For years now, developers have been able to request access to a sandbox on the Test server for development. This has the advantage of working on a system with everything set up already, but the downside of requiring internet connectivity for development and an inability to use GUI development tools. Also, the shared database restricts doing any sort of disruptive changes without impacting others.
In an attempt to address these challenges, we've created a virtual machine (or VM) running 64-bit Ubuntu that contains a fully-configured and functional version of the DP code and all its middleware. It even includes preconfigured DP users (admin, pm, proofer) and a demo project loaded and ready for proofing.
The VM requires 1GB of RAM and 2GB of disk space (although it is thin-provisioned and can grow up to 20GB if necessary).
What this is for
You can use the VM for a couple of different use-cases.
Because the VM is entirely self-contained and includes predefined users and one loaded trial project, it can be used to try out the DP code without installing it yourself.
DP code development
This is its primary use-case. Within the VM you can set up a git environment, make and commit code modifications, and test them out. When you're done you can push your code back to GitHub. Note that for most things you will still need a way for other developers and testers to review and test your code before it can be merged into the main repository. So you probably still want a Test server account if only for that purpose.
Selecting a VM file format
The virtual machine was created using VMware Fusion, and is available both in the VMware format and as an OVA for use in other virtualization products like VirtualBox.
The OVA file can be used many virtualization products, such as:
- VirtualBox - Windows, Linux, Mac (all free)
The VMware VM uses hardware version 9, meaning it can be used with the following VMware products:
- Player 9 or later - Windows and Linux (free for personal use)
- Fusion 5 or later - Mac (must be purchased)
- Workstation 9 or later - Windows and Linux
- ESXi 5.1 or later
Getting the VM
Note that each VM is versioned with its release date both in the zip filename and within the README.md files.
- README file
- v20200905 (latest version) - running R202009 DP release; See #VM_Errata for a required change to use this.
- VMware format (~1.1G)
- shasum: a94002648ff747a2b1737174beb73123270df029
- OVA format (~1.3G)
- shasum: f707f111949fae207960b333a10b3174e5fbf00f
- VMware format (~1.1G)
- v20200207 (older version) - running R202002 DP release
- VMware format (~857M)
- shasum: d89c36196bf9dc857b8b31a86482e15ea2377e25
- OVA format (~951M)
- shasum: 3632b5f0467f7a7fcb976183997ee4301eb33992
- VMware format (~857M)
Installing the VM
Now that you've downloaded the VM, you need to do something with it.
The OVA and VMware VMs can be used in multiple virtualization products but only VMware products and VirtualBox are discussed below.
- Download the OVA (see above)
- Download and install VirtualBox
- Run Virtual Box
- Click File > Import Appliance
- Click the Browse button (a folder with a green chevron in the Windows version) and navigate to the OVA downloaded previously and click Open
- Click Next back on the Import Virtual Appliance screen
- Click Import and the appliance begins to be imported
- In the main VirtualBox window, click the new appliance (probably called "vm") and then Machine > Settings
- You can change the name to something more descriptive. While in the General section, be sure to make note of the contents of the Description tab
- While still in the Settings, click the Network section. By default, the "Attached to" for Adapter 1 will be set to NAT. Click the drop-down and choose Bridged Adapter; this works better for our purposes
- Click OK on the Settings dialog.
- Double-click the icon for the VM to run it.
After you download the VMware VM, you'll need to extract the archive somewhere. You can then use the VMware product to open the VMX file directly.
Using the VM
See the README.md file included in the VMware archive, or the link to it above, for more information about what is included in that VM. If your virtualization software supports snapshots, it is recommended that you snapshot the VM before you start it up so you can roll back to the pristine snapshot if you want to start afresh.
- Start the VM.
- Log in as 'developer' with password 'developer'
- See /home/developer/README.md in the VM for detailed information about how to configure the DP code. It also contains information on the system accounts, DP accounts, and passwords used in the VM.
- After configuring the DP code per the instructions, you should be able to access the site with your web browser. You can also SSH into the box as the 'developer' user.
Tips & Tricks
- To make the VM as small as possible, it does not include a GUI. Feel free to install one.
- If you want to use editors and other development tools on your local workstation, use Shared Folders to configure access to files on the VM directly.
The following errors were discovered in the current VM and will be corrected in the next version.
You must install the php7.4-mbstring package:
sudo apt install -y php7.4-mbstring
This fix must be applied to allow proofreading projects due to a collation change in MySQL 8.0.
diff --git a/pinc/DPDatabase.inc b/pinc/DPDatabase.inc index 39a04df97..6e4014543 100644 --- a/pinc/DPDatabase.inc +++ b/pinc/DPDatabase.inc @@ -125,7 +125,7 @@ final class DPDatabase $table_encoding = $row['TABLE_COLLATION']; mysqli_free_result($result); - return $table_encoding == 'utf8mb4_general_ci'; + return stripos($table_encoding, 'utf8mb4') === 0; } // Prevent this class from being instantiated
No known problems, but if you are experiencing network issues with VirtualBox, see #Troubleshooting below.
Converting the VM to Unicode
The v20200207 VM can be converted to support the in-development Unicode branch fairly easily:
Log into the machine as the 'developer' user. # Pull the latest code cd ~/public_html/dproofreaders/ git fetch origin git checkout origin/unicode # convert the database to UTF-8 echo "ALTER DATABASE dp_db DEFAULT CHARACTER SET utf8mb4;" | mysql -udp_user -pdp_password dp_db # run the upgrade scripts cd ~/public_html/dproofreaders/SETUP/upgrade/14 # Note: these are the upgrade scripts available at the time of this writing, # there may be more. Run all in the directory in alpha order. php 20190818_convert_db_to_utf8mb4.php php 20190819_convert_project_tables_to_utf8mb4.php php 20191211_convert_project_text_files.php php 20191211_convert_word_lists.php php 20200127_add_glyphsets_tables.php # update the charset in configuration.sh sed -i "s/_CHARSET='ISO-8859-1'/_CHARSET='UTF-8'/" ~/configuration.sh # Re-run the configuration script cd /home/developer/public_html/dproofreaders SETUP/configure /home/developer/configuration.sh .
- The VM fails to boot.
- The VM is running a 64-bit version of Ubuntu. To use the VM, both VMware and VirtualBox requires either a 32-bit CPU or 64-bit CPU with hardware virtualization support (VT-x or AMD-V) enabled. Look up your CPU model online to see if it supports this. If it does, ensure that it is enabled in the BIOS.
- I'm unable to log in or I keep getting redirected to invalid pages.
- Be sure to run the configure.sh script per the README.md. The code relies on $code_url being set correctly in site_vars.php. For "real" sites like pgdp.net, this is a static field, but the VM uses the IP address as part of $code_url. This requires that users run configure.sh to correctly set that for their environment. Note: If you run configure.sh and it fails with a message that it couldn't find the ifconfig command, rerun it using sudo, or switch to superuser mode first using "sudo su -".
- I don't have network access inside the VM.
- VirtualBox presents a different network interface than the one VMWare uses and the system is configured for. If the VM comes up without any internet access try the following:
# test to see if the network interface is named enp0s17 sudo dhclient enp0s17 ping -c5 www.google.com # if the ping works, continue with the following steps to update system files to use that interface # if the ping fails, drop cpeel a PM or find him in Slack # update the system to use network interface enp0s17 sudo pico /etc/network/interfaces # in the Primary network stanza, change the two instances of ens33 to enp0s17 # save and exit the file # now update the DP configuration.sh file pico ~dproofreaders/configuration.sh # in the base_url line, change ens33 to enp0s17 # save and exit the file # now reboot the system and re-run the configure script per the README.md
VMware vs VirtualBox vs vagrant
The VM was created using VMware because that's what the creator had available. Instructions for converting it to OVA provided by WalterAM! (thanks WalterAM!).
If some enterprising individual would like to make this into a vagrant config, we'd be delighted to have it.
Steps to release a new developer VM
These are notes generated during the creation and publishing of the VM -- you won't need these to use it.
See the Discussion page for tweaks to decrease the size of the VM by removing unnecessary packages.
Things to remember when setting up the VM with new code:
- Update /home/developer/configuration.sh with the latest version. SETUP/import_old_configuration.php is really useful for this.
- Run all upgrade scripts.
- Ensure a clean 'git status' at the end
- Do a 'git gc' to do a quick garbage cleanup.
After the VM is set up, these were the steps taken to release a new developer VM:
- Ensure the /home/developer/README.md is up-to-date, including the release version (the date it will be released)
- Ensure the VM note reflects the same information as the /home/developer/README.md
- Ditto the README.md on the host that will be included as part of the .zip
- Ensure the OS on the VM is as up-to-date as possible
- Remove any snapshots
- Remove /var/log/apache2/* files
- apt-get clean
- Reboot to rescue and access root shell (see https://wiki.ubuntu.com/RecoveryMode)
- umount /dev/sda1
- zerofree -v /dev/sda1
- service rsyslog stop
- service dhclient stop
- umount /
- zerofree -v /dev/mapper/dpdevel--vg-root
- Reboot into regular boot
- Shrink the VM:
- vmware-toolbox-cmd disk shrink /
- vmware-toolbox-cmd disk shrink /boot
- Zero out swap
- swapoff --all
- dd if=/dev/zero bs=4k of=/dev/mapper/dpdevel--vg-swap_1
- mkswap /dev/mapper/dpdevel--vg-swap_1
- Shut down the VM
- Create a new directory on the host named like dp-devel-YYYYMMDD
- Put the README.md in it and a copy of the VM
- Zip up the directory like this: zip -r -9 dp-devel-VM-YYYYMMDD.zip dp-devel-VM-YYYYMMDD/
- Use the ovftool to create an OVA from the VMware VM
- ovftool /path/to/dp-devel.vmx /path/to/dp-devel.ova
- Upload README.md, the .zip and the .ova to www.pgdp.org:/data/htdocs/dp-devel-VM
- Rename /data/htdocs/dp-devel-VM/README.md with the release date and update the README.md symlink to point to it
- Update #Getting the VM and #VM Errata as necessary, probably removing the oldest image and just keeping the two most recent ones.