GISTEMP: Build Notes

I received an email request to assist in a linux build of GISTEMP. Looking at my old notes and directories, I realized that I did not have an end-to-end instruction set. And as far as I can tell, no one else does either. So here they are. The build has become much easier, primarily due to bug fixes over the last 2 years or so, many of them initiated by findings from Barnes and Jones at Clear Climate Code.

---

Install Oracle Virtual Box

I’m beginning with the instructions to load Oracle Virtual Box (formerly of Sun). If you already have a VM, or have a stand-alone Ubuntu install, you can of course skip these steps. There is an Extension Pack and Guest Additions available for the Virtual Box which makes it easier to work between the host and VM systems, but they are not included in this instruction set. If you are new to Oracle VirtualBox, be sure to learn more about these.

#===========================================
# Install Oracle Virtual Box
#===========================================

In a browswer, go to:

http://www.virtualbox.org/wiki/Downloads

Download VirtualBox 4.0.4 for Windows hosts

http://download.virtualbox.org/virtualbox/4.0.4/VirtualBox-4.0.4-70112-Win.exe

Double click on VirtualBox-4.0.4-70112-Win.exe
(Note: I had a previous version installed on my system,
some dialogs may differ on a fresh installation)

In the ‘Oracle VM VirtualBox 4.0.4 Setup’ window,
Select ‘Next’

In the ‘Custom Setup’ dialog,
Select ‘Next’

In the ‘Custom Setup’ dialog
Select the options that you would like
Select ‘Next’

In the ‘Warning: Network Interfaces’ dialog,
Select ‘Yes’

In the ‘Ready to Install’ dialog,
Select ‘Install’

(Note: I have excluded mentioning
the Windows 7 security related dialogs)

In the ‘Oracle VM VirtualBox 4.0.4 Setup’ window,
Select ‘Finish’

---

Configure Ubuntu 10.10 VM

Deploying a new VM is very simple. I found that the default memory configuration of 512MB will cause a “out of memory” error in GISTEMP STEP 5. So set that to a minimum of 1024 MB. Otherwise, I accept the defaults. The network settings below work for my host environment. Yours might differ.

#===========================================
# Configure Ubuntu 10.10 VM
#===========================================

Start Oracle VM Virtual Box (Version 3.2.6.r63112)
Select ‘New’

In the “Welcome to the New Virtual Machine Wizard” dialog,
select ‘Next’

In the “VM Name and OS Type” dialog,
Enter: ‘ub1010-gistemp’
Select ‘Linux’
Select ‘Ubuntu’
Select ‘Next’

In the ‘Memory’ dialog,
Select ’1024MB’
Select ‘Next’

In the ‘Virutal Hard Disk’ dialog,
Select ‘Next’

In the ‘Welcome to the Create New Virtual Disk Wizard’ dialog,
Select ‘Next’

In the ‘Hard Disk Storage Type’ dialog,
Select ‘Next’

In the ‘Virtual Disk Location and Size’ dialog,
Select ‘Next’

In the ‘Summary’ dialog,
Select ‘Finish’

In the ‘Summary’ dialog,
Select ‘Finish’

In the ‘Oracle VM VirtualBox’ window,
Right Click on ‘ub1010-gistemp’
Select ‘Settings…’

In the ‘ub1010-gistemp’ window,
In the left hand column,
Select ‘Network’

In the ‘Adapter 1′ tab,
Select ‘Bridged Apdapter;
Select ‘DW1501 Wireless-N WLAN Half-Mini Card’
(The above setting is machine dependent)
Select ‘OK’

---

Initiate Ubuntu 10.10 Desktop Installation

Download and load the Ubuntu 10.10 Desktop image.

#===========================================
# Initiate Ubuntu 10.10 Desktop Installation
#===========================================

In a browser, go to:

http://www.ubuntu.com/desktop/get-ubuntu/download

Download Ubuntu 10.10 Desktop:
ubuntu-10.10-desktop-i386.iso

If not already started,
Start the “Oracle VM VirtualBox Manager”

In the left hand column,
Highlight ‘ub1010-gistemp’

In the upper menu,
Select ‘Start;

In the ‘First Run Wizard’ dialog,
Select ‘Next’

In the ‘Select Installation Media’ dialog,
Select the small ‘directoy’ icon in the “Media Source’ panel.

In the “Choose a virtual CD/DVD disk file” window,
Navigate to the directory you downloaded the Ubuntu image.
C:\Users\someone\Downloads\ubuntu-10.10-desktop-i386.iso
Select ‘Open’

In the ‘Select Installation Media’ dialog,
Select ‘Next’

In the ‘Summary’ dialog,
Select ‘Finish’

The ‘ub1010-gistemp [Running] – Oracle VMBox’ window opens.

---

Install Ubuntu 10.10 Desktop

Install the Ubuntu 10.10 Desktop operating system into the VM.

#===========================================
# Install Ubuntu 10.10 Desktop
#===========================================

In the ‘ub1010-gistemp [Running] – Oracle VMBox’ window …

In the ‘Install’ window,
Select ‘Install Ubuntu’

In the ‘Preparing to install Ubuntu’ dialog,
Select ‘Forward’

In the ‘Allocate drive space’ dialog,
Select ‘Forward’

In the ‘Allocate drive space’ dialog,
Select ‘Install Now’

In the ‘Where are you?’ dialog,
Select ‘Forward’

In the ‘Keyboard layout’ dialog,
Select ‘Forward’

In the ‘Who are you’ dialog,
In the ‘Your name’ field, enter:
In the ‘Your computer’s name’ field, enter:
In the ‘Pick a username, enter:
In the ‘Choose a password, enter:
In the ‘Confirm your password, enter:
Select ‘Forward’

In the ‘ub1010-gistemp [Running] – Oracle VMBox’ window,
In the upper menu,
Select Devices -> CD/DVD Devices -> Remove Disk from Virtual Drive

In the ‘Installation Complete’ window,
Select ‘Restart Now’

---

Update Ubuntu 10.10 Desktop

Update the Desktop install. I had 300 meg of updates, so expect this to take a while.

#===========================================
# Update Ubuntu 10.10 Desktop
#===========================================

In the ‘ub1010-gistemp [Running] – Oracle VMBox’ window …

In the login window, select
In the password field, enter:
Select ‘Log In’

From the Ubuntu Desktop Dropdown Menu
Select System -> Administration -> Update Manager

In the ‘Update Manager’ window,
Select ‘Install Updates’

In the ‘Authenticate’ dialog,
Enter

Note: This will take some time
When update is completed …

In the ‘Update Manager’ window,
Select ‘Restart Now’

After the VM has restarted,
In the lower right hand tool bar,
Select -> Shutdown

---

Snapshot ub1010-giss VM

Snapshots allow you to restore your VM back to an earlier state. Just after a fresh install and update is a good place to capture a snapshot.

#===========================================
# Snapshot ub1010-giss VM
#===========================================

In the “Oracle VM VirtualBox Manager” window …

In the left hand column,
Highlight ‘ub1010-gistemp’

In the upper right hand menu
Select ‘Snapshots’

In the right hand panel,
Select the ‘Snapshot’ icon (camera)

In the ‘Take Snapshot of Virtual Machine’ dialog,
In the ‘Snapshot Name’ field, enter: Fresh Install
In the ‘Snapshot Enter’ field, enter: Bare Metal Install and Update
Select ‘OK’

---

Install required packages

Install gfortran and ksh to support the GISTEMP scripts and compile the fortran.

#===========================================
# Install required packages
#===========================================

In the “Oracle VM VirtualBox Manager” window …

In the left hand column,
Highlight ‘ub1010-gistemp’

In the upper window,
Select ‘Start’

In the login window, select
In the password field, enter:
Select ‘Log In’

From the Ubuntu Desktop Menu,
Select Applications -> Accessories -> Terminal

In the ‘@:~’ terminal window,
Enter: sudo apt-get install gfortran
Enter:
Enter: Y

In the ‘@:~’ terminal window,
Enter: sudo apt-get install ksh

#(Note: The following is used to mount my shared files.
# I include it only for completeness)
# In the ‘@:~’ terminal window,
# Enter: sudo apt-get install smbfs
# mkdir ~/mybook
# alias mc=’echo | sudo smbmount “//192.168.0.92/my book” mybook’

---

Install required software

The Python package installed by Ubuntu does not include all the module scripts required by the GISTEMP python Makefiles. This can be worked around by compiling and installing a Python package from the upstream source.

#===========================================
# Install required software
#===========================================

#- The Ubuntu Python package does not include
#- everything needed to build older python modules

In the ‘@:~’ terminal window,
Enter: sudo su
Enter: mkdir /opt/tmp
Enter: mkdir /opt/python-2.6
Enter: mkdir /opt/gistemp-linux
Enter: chown : /opt/tmp
Enter: chown : /opt/python-2.6
Enter: chown : /opt/gistemp-linux
Enter: exit

From the Desktop Toolbar,
Select ‘Firefox’
Navitge to:

http://www.python.org/download/releases/2.6.6/

Download the ‘Gzipped source tar ball’ to your Downloads directory

http://www.python.org/ftp/python/2.6.6/Python-2.6.6.tgz

In the ‘@:~’ terminal window,
Enter: cp ~/Downloads/Python-2.6.6.tgz /opt/tmp
Enter: cd /opt/tmp
Enter: tar -xvzf Python-2.6.6.tgz
Enter: cd Python-2.6.6
Enter: ./configure –prefix=/opt/python-2.6
Enter: make
Enter: make install

---

Download GISTEMP, data, and helper scripts

We now have an environment ready to support the GISTEMP installation. Download the source, data, and helper scripts.

#===========================================
# Download GISTEMP, data, and helper scripts
#===========================================

#- Use the command ‘wget’ to save these files to the Downloads directory
Enter: cd ~/Downloads
Enter: wget ftp://data.giss.nasa.gov/gistemp/sources/GISTEMP_sources.tar.gz
Enter: wget ftp://ftp.ncdc.noaa.gov/pub/data/ghcn/v2/v2.mean.Z
Enter: wget
ftp://ftp.ncdc.noaa.gov/pub/data/ushcn/v2/monthly/9641C_201103_F52.avg.gz
Enter: wget ftp://data.giss.nasa.gov/gistemp/SBBX.HadR2
Enter: wget http://rhinohide.org/gw/code/gistemp/scripts/swap.f
Enter: wget

http://rhinohide.org/gw/code/gistemp/scripts/Makefile.rb.monthlydata

Enter: wget

http://rhinohide.org/gw/code/gistemp/scripts/Makefile.rb.stationstring

In the ‘@:~’ terminal window,
Enter: cd /opt/gistemp-linux
Enter: tar xvzf ~/Downloads/GISTEMP_sources.tar.gz

---

GISTEMP STEP0

Step 0 combines data from several sources and produces a file (v2.mean_comb) similar to the GHCN v2.mean file from NOAA/NCDC. You can learn more about the folowing steps from the GISTEMP sources page.

#===========================================
# GISTEMP STEP0
#===========================================

Enter: cd /opt/gistemp-linux/GISTEMP_sources/STEP0

#- install data files not provided
Enter: cd input_files
Enter: cp ~/Downloads/v2.mean.Z .
Enter: uncompress v2.mean.Z
Enter: cp ~/Downloads/9641C_201103_F52.avg.gz .
Enter: gunzip 9641C_201103_F52.avg.gz
Enter: cp 9641C_201103_F52.avg USHCNv2.avg
Enter: cd ..

#- correct run script for ‘tail’ syntax
Enter: cp do_comb_step0.sh do_comb_step0.sh.orig
Enter: sed ‘s/+100/-100/’ do_comb_step0.sh.orig > do_comb_step0.sh

#- run step 0
Enter: export PATH=.:$PATH
Enter: export FC=gfortran
Enter: do_comb_step0.sh v2.mean

#- copy output
Enter: mkdir ../STEP1/temp_files
Enter: cp temp_files/* ../STEP1/temp_files
Enter: cd ..

---

GISTEMP STEP1

Step 1 provides combination of different station records at a single location and removes of stations identified through manual inspection at some point eariler.

#===========================================
# GISTEMP STEP1
#===========================================

Enter: cd STEP1
Enter: mkdir work_files

#- compile python stationstringmodule.so
Enter: cd EXTENSIONS/stationstring
Enter: cp ~/Downloads/Makefile.rb.stationstring .
Enter: make -f Makefile.rb.stationstring
Enter: cp stationstringmodule.so ../..

#- compile python monthlydatamodule.so
Enter: cd ../monthlydata
Enter: cp ~/Downloads/Makefile.rb.monthlydata .
Enter: make -f Makefile.rb.monthlydata
Enter: cp monthlydatamodule.so ../..

# run step 1
Enter: cd ../..
Enter: do_comb_step1.sh

# copy output
Enter: mkdir ../STEP2/temp_files
Enter: cp temp_files/Ts.txt ../STEP2/temp_files
Enter: cd ..

---

GISTEMP STEP2

Step 2 splits the data into 6 zones for processing. Stations with less than 20 years of data are dropped. And it creates two sets of files: one with periurban adjustements and one without.

#===========================================
# GISTEMP STEP2
#===========================================

Enter: cd STEP2
Enter: mkdir work_files

#- run step 2
Enter: do_comb_step2.sh 2011

# copy output
Enter: mkdir ../STEP3/temp_files
Enter: cp temp_files/Ts.GHCN.CL.* ../STEP3/temp_files
Enter: cd ..

---

GISTEMP STEP3

The data is gridded and Land Anomaly tables are created.

#===========================================
# GISTEMP STEP3
#===========================================

Enter: cd STEP3

#- run step 3
Enter: do_comb_step3.sh

#- display GISTEMP Land Anomalies
Enter: cat results/GLB.Ts.GHCN.CL.PA.txt

# You can compare results with
# http://data.giss.nasa.gov/gistemp/tabledata/GLB.Ts.txt

# copy output
Enter: cp temp_files/SBBX1880.Ts.GHCN.CL.PA.1200 ../STEP4_5/input_files
Enter: cd ..

---

GISTEMP STEP4_5

Step 4 is used to add the latest OI SST data to the Had.R2 file of SST temperatures.
Step 5 combines the Land data with the SST data.

I don’t recall when/where I found the swap.f file. I am sure that it came from a GISS ftp site, but since I cannot relocate it, I’m offering a copy on my site.

#===========================================
# GISTEMP STEP4_5
#===========================================

#- there is code here to update SBBX.HadR2 files
#- with recent OI data, which I forego for the moment

Enter: cd STEP4_5

#- convert byte order of file
Enter: cp ~/Downloads/swap.f .
Enter: gfortran swap.f -o swap.exe
Enter: cp input_files/SBBX1880.Ts.GHCN.CL.PA.1200
input_files/SBBX1880.Ts.GHCN.CL.PL.1200-orig
Enter: ln -s input_files/SBBX1880.Ts.GHCN.CL.PL.1200-orig DAT
Enter: swap.exe
Enter: mv DAT_REV input_files/SBBX1880.Ts.GHCN.CL.PL.1200
Enter: rm DAT

#- run step 5
Enter: export GFORTRAN_CONVERT_UNIT=BIG_ENDIAN
Enter: do_comb_step5.sh

#- display GISTEMP Land and Sea Anomalies
Enter: cat results/GLB.Ts.ho2.GHCN.CL.PA.txt

# You can compare results with
# http://data.giss.nasa.gov/gistemp/tabledata/GLB.Ts+dSST.txt

You should now have a working set of GISTEMP scripts and executables. There is more to learn on managing the input files and displaying the output.

---

Instructions are here: ub1010-gistemp.txt

About these ads