DP Code Development Using git

From DPWiki
Jump to: navigation, search

DP source code is hosted on GitHub. This page discusses the git workflow adopted by DP and offers some assistance to developers familiar with other source control systems (CVS, SVN) but who are new to git. See also: git crash course for DP developers.

In March 2018 the git repo was moved from SourceForge to GitHub. See DP Code Migration to GitHub for instructions on how to migrate your repos to github.com.

git development model

There are multiple ways to use git to develop. DP has adopted a "fork, branch, develop, merge" model, where developers are expected to:

  1. fork the main repository
  2. create a development branch off master
  3. develop code on the branch
  4. generate a merge/pull request when ready to merge

Initial repository set up

To get started with the git repo for DP, you have to first create a fork of the code repository under your GitHub user ID. You only need to do this once ever.

  1. If you haven't yet, create a GitHub account.
  2. Log into your GitHub account.
  3. Add your public SSH keys to your GitHub account if you haven't done so already. Be sure that the matching private key is installed on the system from which you will be running the git commands.
  4. Go to the dproofreaders repo and fork the repository by clicking Fork in the upper right
    • Click on your github username as the location to create the fork. Note that at the bottom of the popup there will be a message that seems to say you don't have authority to create a fork. But what it really says is that you can't create your fork in *dproofreaders*.
  5. After a few seconds, this will take you to your forked copy of the dproofreaders repo under your username

This will create a fork of the code under your GitHub userid, eg: https://github.com/cpeel/dproofreaders

Workflow 1: All on a test machine

The following workflow assumes your development is being done on a machine where you can fully test the changes you are making as you make them. That may be your own workstation running Apache, mysql, and PHP, a copy of the DP Code Development VM, or the Test server.

Note: Your workstation must be using the Supported middleware versions in order to be useful as a development platform. If it isn't, you are strongly encouraged to use either the DP Code Development VM or the Test server.

Workstation set up

After the repository is set up, you need to set up the code on your workstation. You only need to do this once per development system.

  1. On your workstation (e.g., your home computer), open a shell/terminal window.
  2. If this is your first time using git on this system, start by configuring git with your name and email address. Note that these values will be visible publicly:
    • git config --global user.name "<NAME>"
    • git config --global user.email "<EMAIL>"
      • Consider using <USERNAME>@users.noreply.github.com if you don't want your personal email address publicly accessible.
  3. Go to the directory/folder in which you want to do DP development. You probably want this to be web-accessible.
    • If using the Test server, create the $HOME/public_html directory and use that:
      mkdir -p $HOME/public_html
      cd $HOME/public_html
    • If using the DP Code Development VM, the $HOME/public_html directory should already exist if you followed the instructions at the end of the README.txt file. If it doesn't, create it.
  4. In a browser, go to your fork of the code in GitHub.
  5. Click the green "Clone or download" button and copy the URL it lists there and run git clone $URL in the same shell from step 1.
  6. Because the command tries to establish an SSH connection to GitHub, you may get some SSH warnings, e.g. "The authenticity of host 'git@github.com' can't be established", and your OS may ask you to supply the password associated with the public key that GitHub has for you.
  7. Depending on your connection speed, the command will take 15 seconds or so to pull down a clone of your repository. When done, it will have created a single sub-directory, named e.g. dproofreaders -- this is referred to as <checkout_name> in step 12 below.
  8. cd into that directory, then run the following command to add the primary DP repository as an 'upstream remote' for your local clone:
    git remote add upstream https://github.com/DistributedProofreaders/dproofreaders.git
  9. Create a DP configuration file by following the directions in SETUP/configuration.sh. This file will be denoted <config.sh> below.
    • If using the Test server, use ~cpeel/bin/TEST_setup.sh.
    • If using the DP Code Development VM, use /home/developer/configuration.sh.
    • Both configuration files above assume the code is web accessible at http://HOSTNAME/~$USER/c, so you need to either create the symlink mentioned below or change your copy of the configuration file.
  10. Configure the code so it is web-accessible by going to the base checkout directory (eg: in u-cpeel-dp) and running:
    SETUP/configure <config.sh> `pwd`
    This will create a locally-configured file for each .template file.
  11. (optional) Create a symlink so that the code is web-accessible from ~username/c instead of ~username/<checkout_name>:
    ln -s $HOME/public_html/<checkout_name>/ $HOME/public_html/c

Developing

After you have set up your development environment, you can develop!

Note that all of these steps only make changes in your fork, not the main repository, so feel free to play with these commands all you want -- you can't break anything.

  1. Create a branch to capture the work you are doing, use a branch name that is descriptive but concise
    git branch <branchname>
    git checkout <branchname>
  2. To configure the code so it is web-accessible, change directories to the checkout directory (eg: in u-cpeel-dp) and run:
    SETUP/configure <config.sh> .
    This will create a locally-configured file for each .template file.
  3. Make any changes you want using a standard git workflow:
    1. Make code changes and test them
    2. Group logical changes into one commit via:
      git add file1 file2 file3
      git add file4
      git commit
      • In the commit message, the first line should be 50 characters or fewer and summarize the commit; then a blank line; then lines of 72 characters or fewer with additional details.
    3. Repeat 1 & 2 as needed
  4. To push your changes up to GitHub for others to see:
    git push origin <branchname>

Workflow 2: Develop on one machine, test on another

If you primarily develop on one machine but test on another, such as developing on a machine with a GUI and copying your code over to a separate server with the web server and database (such as the Test server), the following workflow is for you.

In the instructions below, Development system is the system where you will be editing source code. Test system is the system running Apache, PHP, and mysql where you test your changes and will commit the code. The important part of this workflow is that while both systems have read access to the remote git repo, only the Test system has write access. This is to reduce the possibility of introducing merge conflicts when a branch is updated from two different sources.

Note: The test system must be using the Supported middleware versions in order to be useful for testing. If it isn't, you are strongly encouraged to use either the DP Code Development VM or the Test server for that purpose.

Test system set up

After the repository is set up, you need to make the code accessible on the test system. This is the system where you will make all commits. You only need to do this once per test system.

  1. On the Test system, open a shell/terminal window.
  2. Follow steps 2 and later in the #Workstation set up section.

Development system set up

This will make the code accessible on your development system. You only need to do this once per development system.

  1. On the Development system (e.g., your home computer), open a shell/terminal window, and go to the directory/folder in which you want to do DP development.
  2. In a browser, go to your fork of the code in GitHub.
  3. Click the green "Clone or download" button and copy the URL it lists there and run git clone $URL in the same shell from step 1.
    • This will clone the repository and check out the most recent code in the master branch.
  4. Depending on your connection speed, the command will take 15 seconds or so to pull down a clone of your repository. When done, it will have created a single sub-directory, named e.g. u-cpeel-dp.

Developing

After you have set up the Test system and Development system, you can develop!

Note that all of these steps only make changes in your fork, not the main repository, so feel free to play with these commands all you want -- you can't break anything.

  1. On the Test system create a branch to capture the work you are doing, use a branch name that is descriptive but concise
    git branch <branchname>
    git checkout <branchname>
  2. On the Development system make any changes you want
  3. Copy them to the Test system to test them
  4. After the changes test OK, commit them on the Test system:
    1. Group logical changes into one commit via:
      git add file1 file2 file3
      git add file4
      git commit
      • In the commit message, the first line should be 50 characters or fewer and summarize the commit; then a blank line; then lines of 72 characters or fewer with additional details.
  5. Repeat 2-4 as needed
  6. To push your changes up to GitHub for others to see, from the Test system:
    git push origin <branchname>
  7. To refresh your Development system code from what is in your branch:
    1. Fetch changes (this will not change the code you have checked out):
      git fetch origin
    2. See what files will be overwritten by the refresh:
      git status
      Ensure they have been committed into the branch on the Test system.
    3. Forcibly check out the branch, this will clear out any changes in your code base that isn't checked in somewhere else!:
      git checkout -f <branchname>

Refreshing your fork

The following walks you through refreshing the code in your fork to include updates since you forked. You will probably always want to update the master branch in your repo. If you have one or more code branches you're working on, you can update those too. These steps are not required to open a merge request.

Note, if using Workflow #2, all of them are done from the Test system.

  1. To refresh your fork from the project master on GitHub (ie: the upstream repo):
    git checkout master
    git fetch upstream
    git rebase upstream/master
  2. To refresh a code branch (if you have one):
    git checkout <branchname>
    git rebase master
  3. And then push them back up with:
    git push origin master
    git push origin <branchname>
    • If you've pushed your branch to origin and then rebase it (which is all OK), you've altered the commit stream and the push will fail with the message:
      To prevent you from losing history, non-fast-forward updates were rejected
      Merge the remote changes (e.g. 'git pull') before pushing again.


      If this happens, you can force the push such that the local changes overwrite the changes in origin:
      git push origin --force <branchname>
  4. It's always a good idea to re-run the configure script after refreshing your fork in case changes have been made in pinc/site_vars.php.template
    # note the trailing period!
    SETUP/configure <config.sh> .

Publishing your changes in a separate sandbox

After you have code for others to look at, it's often useful to push the changes to a separate directory so you can continue developing in your main sandbox. You can use ~cpeel/bin/publish-branch.sh to do this on the TEST server.

  1. Check out your branch
  2. Run ~cpeel/bin/publish-branch.sh

This will create a new directory based on the current branch name and copy the current checkout contents to the new directory. This includes not only code in your branch but also any files that are in the directory but not checked in.

To delete the published directory after you are done, you can either manually remove the directory under public_html/c.branch or check out the branch you want to delete and run ~cpeel/bin/unpublish-branch.sh

Merging your changes

After you have code you want merged into the main DP repository, open a pull request from your fork on GitHub so devs can review your code.

  1. Go to your fork in GitHub (eg: https://github.com/cpeel/dproofreaders)
  2. Click "New pull request" to generate a pull request
  3. Select the branch you want to merge from the 'compare:' drop-down on the right
    • This will show you the commits that will be included in the pull request and the changes in those files
  4. Click "Create pull request"
    • The Title should contain a short summary of what your branch has in it
    • The Comment field should contain a more lengthy description of what your changes do.
  5. Click "Create pull request" to finalize the pull request

If, as a result of subsequent code review, you add some commits to your branch, and push them to your fork on GitHub, your pull request will be updated automatically.

Reviewing code

There are multiple ways to review code, either that is just in another user's fork or that they've submitted in a merge request. Different people will want to do it differently, but here are some ideas:

To access the other user's code:

  • You can view the individual commits and the entire tree via the GitHub web UI.
  • Set up the other user's repo as a remote in your local repo, then fetch their repo, check out their branch, and review the code:
    git remote add user_blah https://github.com/cpeel/dproofreaders.git
    git fetch user_blah
    git checkout user_blah/branchname
  • Clone the other user's repo as an entirely new local repo, then check out their branch and review the code:
    git clone https://github.com/cpeel/dproofreaders.git
    git checkout branchname

If you opt to have the code locally, the following are useful git commands after you have checked out their branch:

  • Review the commit log for the current branch that is checked out:
    git log
  • Review the commit log for an arbitrary branch:
    git log <branchname>
  • View the diff between two commits:
    git diff <hash1> <hash2>

git Training Resources