TRIUMF Tier-1 Recipes > Table of Contents

Menu Name / SubjectAuthorEntry Date
Analytics
Elasticsearch
Adding Missing Object .......................................fernandoSep 01 2021
Copying a Field Into Another ................................fernandoJan 11 2021
Creating a New Rolling Index ................................fernandoJan 11 2021
Creating a Snapshot .........................................fernandoJan 11 2021
Deleting a Field ............................................fernandoJan 11 2021
Forcing an Index Rollover ...................................fernandoJan 11 2021
Increasing Search Fields ....................................fernandoJan 11 2021
Re-indexing .................................................fernandoJan 11 2021
Renaming a Field ............................................fernandoJan 11 2021
Rolling Upgrade of Elasticsearch Nodes ......................fernandoJan 11 2021
Unassigned/Corrput Shard Troubleshooting ....................fernandoApr 28 2021
Update Field ................................................fernandoFeb 16 2023
Logstash
Batch System
HTCondor connection brokering (CCB) .........................dqingSep 27 2018
Limit maximum number of running jobs for certain u ..........dqingJun 07 2016
Modifying the queue status of a worker node .................deatrichOct 07 2013
Putting workers on or off-line ..............................deatrichJul 23 2014
Reassign WNs between different Torque servers ...............dqingJun 01 2009
Shuffling out-of-warranty IBM blade hardware ................deatrichApr 06 2018
Slow/failing response to pbs commands .......................deatrichOct 15 2008
Torque Job Submission Filter ................................dqingMar 14 2011
Using mcfloat for dynamic partitions in Torque for ..........dqingMay 15 2015
Cloud
Atlas Federation - CEPH S3 - (Removed - Reference ..........fernandoJul 11 2017
Atlas Federation Infrastructure .............................fernandoJun 20 2017
Atlas Federation Infrastructure - Administration ............fernandoJan 12 2018
Atlas Federation Infrastructure - Checksums .................fernandoSep 05 2019
Atlas Federation Infrastructure - Shutdown Procedu ..........fernandoOct 27 2017
Openstack Commands ..........................................aywongDec 20 2013
Openstack Installation ......................................aywongDec 18 2013
Database
MySQL
Auto clean up of MySQL Replication Logs .....................aywongOct 22 2019
Benchmark MySQL Installation ................................aywongOct 11 2013
Changing innodb config and starting replication .............deatrichSep 15 2010
Clean up of replication logs ................................deatrichNov 12 2010
Cleaning out master logs on the LFC .........................deatrichDec 05 2008
Example mysql stored procedure ..............................deatrichNov 15 2010
Implement Hash Security .....................................aywongOct 11 2013
Mariadb - Master_Master replication on nms1 & nms2 ..........aywongJul 23 2021
MySQL - mariadb comands .....................................aywongApr 28 2017
MySQL - set parameter expire_logs_days ......................aywongFeb 14 2018
MySQL tidbits ...............................................deatrichSep 23 2008
Quick way to import old mysql database to mariadb ...........deatrichNov 21 2018
Repopulate a MySQL Slave ....................................aywongJan 26 2012
Securing a default, fresh install of MySQL ..................deatrichSep 23 2008
Starting another mysql replica slave ........................deatrichApr 22 2009
Oracle
ATLAS 3D Oracle Startup & Shutdown Procedure ................aywongOct 28 2013
ATLAS FroNTier Service ......................................aywongOct 28 2013
Deploying RHEL for an Oracle RAC node .......................deatrichJun 26 2014
Interpreting OEM warnings: distinct types of ORA- ..........deatrichApr 16 2009
Manually move the SCAN_LISTENER on orac1 or orac2 ...........aywongJun 24 2009
oem.lcg.triumf - Oracle Enterprise Manager Service ..........aywongOct 15 2013
Oracle Clusterware - Using srvctl commands ..................aywongAug 22 2014
Oracle SCAN LISTENER Commands ...............................aywongOct 15 2013
RMAN - Database ID ..........................................aywongOct 17 2008
Starting and Stopping OEM Agents on Oracle Servers ..........aywongOct 15 2008
Using the gqlplus RPM .......................................deatrichJan 19 2009
RAC Issues
PostgreSQL
Configuring Table Level Autovacuum ..........................aywongJul 16 2013
Create Tablespace ...........................................aywongDec 03 2014
PostgreSQL - SQL ............................................aywongOct 07 2013
Restore a PostgreSQL database from a pg_dumpall .............aywongOct 07 2009
Slony - replicating one database ............................aywongJan 08 2009
dCache Storage
dcache admin nodes backup/recovery procedure ................simonMar 02 2011
dccp stress test scripts on chimera .........................simonApr 17 2009
Space relationship in dCache SrmSpaceManager ................simonOct 15 2009
Standard Storage Throughput Reports .........................deatrichApr 16 2015
Standard Storage Throughput Testing .........................deatrichApr 16 2015
standard tests for new kernel or modules ....................cpayneFeb 18 2009
xrootd proxy server setup ...................................simonJun 20 2014
Adapters
Reloading netxen driver .....................................deatrichMar 09 2015
Controllers
DDN SFA Controller and VM access ............................deatrichSep 12 2014
DDN SFA Controller Miscellaneous Commands ...................deatrichSep 12 2014
DDN SFA Controller Quick Start Guide ........................deatrichSep 12 2014
Disk
XFS Issues
Running xfs_repair on disabled dCache pools .................deatrichSep 24 2008
Tape
How to check cleaning tape remaining times ..................simonJun 09 2009
Grid Services
Services on tier1nfs ........................................deatrichJun 20 2011
AutoPilotFactory2 - APF2
AutoPyFactory Installation and HowTo ........................desilvaApr 06 2017
CEs
CPU accounting data (ARC CE plus HTCondor batch sy ..........dqingDec 05 2019
Installing ARC CE and HTCondor head node with Ansi ..........dqingNov 13 2017
Condor
Clean WorkerNode ............................................aywongOct 15 2019
Condor - exclude workernode(s) ..............................aywongNov 27 2020
Condor - remove .............................................aywongNov 04 2020
Condor Commands .............................................aywongDec 21 2015
HTCondor Warming Jobs on CE's ...............................aywongApr 19 2016
installing a tarball version of condor ......................desilvaMay 14 2012
Monthly CE Accounting Data ..................................aywongNov 09 2020
Update HTCondor .............................................aywongApr 13 2016
DDM Site Services
Notes on accessing TRIUMF DDM Site Services at CER ..........deatrichOct 14 2008
Frontier
Installing frontier-squid2 when frontier-squid is ..........deatrichFeb 12 2015
Update frontier-squid .......................................aywongNov 30 2021
Upgrade Apache Tomcat on Frontier ...........................aywongMar 02 2011
FTS
Check if FTS is running .....................................aywongAug 13 2009
FTS channel commands ........................................deatrichOct 29 2008
Grid Certificates
Apply New CA Packages to ALL Nodes ..........................aywongOct 29 2015
Bulk renewal of host certificates ...........................dqingJun 15 2010
Locations of yaim copies of host certificates ...............deatrichOct 22 2008
Requesting Host Grid Certificates ...........................deatrichMay 07 2009
Using t1-manage_proxy_functions.sh with grid proxi ..........deatrichFeb 18 2009
LFC
Squids
Quickly restarting squid - clean cache ......................deatrichNov 13 2013
Yaim Configuration
Yaim configuration directives ...............................deatrichDec 04 2008
Yaim configuration process ..................................deatrichDec 04 2008
Hardware
DDN SFA Service call ........................................rajandevSep 22 2015
DDN SFA12K-20E enclosure disk replacement procedur ..........rajandevDec 22 2015
Dell Service call ...........................................rajandevSep 22 2015
How to collect support data for storage machines ............rajandevAug 03 2016
IBM/Lenovo Service call .....................................rajandevSep 22 2015
Latham ......................................................rajandevSep 22 2015
Liebert/Vertiv - UPS Service call ...........................rajandevSep 22 2015
Rooftop AC unit AC9 contact person .........................rajandevDec 12 2017
Running DSA tool on ibm servers .............................deatrichAug 16 2012
Running the explorer tool on Solaris servers ................deatrichFeb 13 2014
Service Contacts ............................................nicoleDec 08 2009
Blades
BIOS changes after replacing CMOS battery of DELL ..........dqingJun 11 2021
Procedure for Blade replacement .............................cpayneDec 02 2008
Disks
Add a new logical drive on IBM ServeRAID 8k .................deatrichMay 01 2009
Checking disk status on Adaptec Raid controllers ............deatrichMar 19 2010
Hot Swapping a SCSI disk ....................................cpayneMar 10 2009
Placing an IBM service call for a bad disk ..................deatrichSep 08 2010
Reseating IBM x3650 disks ...................................deatrichSep 07 2010
Summary of disk performance measurements ....................deatrichOct 10 2008
Using MegaCli ...............................................deatrichFeb 15 2011
Firmware
Getting Dell bios, firmware or drivers ......................deatrichSep 16 2015
Setting up the latest DSA IBM utility .......................deatrichAug 05 2010
Updating firmware on NetXen 10g ethernet cards ..............deatrichMar 12 2014
Updating ILOM firmware and bios on spools ...................deatrichFeb 03 2011
Using the ASU utility to change power restore poli ..........deatrichMar 03 2015
UXSPI
Getting the IBM Machine Type ................................deatrichFeb 26 2014
Updating the local mirror of IBM firmware ...................deatrichFeb 26 2014
Lights-Out
Resetting the management interface on APC powerbar ..........deatrichAug 12 2010
Dell
Commands to configure BIOS and iDRAC on DELL R640 ..........dqingApr 08 2021
Commands to configure BIOS on PowerEdge C6320 ...............deatrichOct 23 2018
Commands to configure BIOS on PowerEdge C6420 ...............deatrichOct 23 2018
Getting a console via ssh to iDRAC interface ................deatrichMar 10 2017
Requesting a signed cert. for Dell remote interfac ..........deatrichMar 25 2011
Resetting DRAC-3 interface on Dell chassis ..................deatrichJan 27 2010
Updating lights-out firmware on a Dell 730 ..................deatrichSep 16 2015
Using racadm via ssh ........................................deatrichMar 07 2017
HP Servers
IBM-Lenovo
Getting a console via ssh to an RSA interface ...............deatrichOct 20 2017
Getting a console via ssh to IBM chassis ....................deatrichMay 10 2017
Getting a console via ssh to IMM2 interface .................deatrichApr 11 2017
IBM IMM SNMP trap configuration .............................vitaliykDec 16 2013
In-band resetting of the XClarity controller ................deatrichApr 29 2019
Lenovo SR630 BIOS Setup .....................................deatrichFeb 09 2018
Lenovo XClarity Setup .......................................deatrichFeb 09 2018
Power on/off an IBM blade via the CLI .......................deatrichJun 22 2017
Reboot IMM interface from ASU ...............................deatrichMar 10 2017
Reboot RSA interface from ASU ...............................deatrichOct 26 2016
Requesting a signed cert. for IBM remote interface ..........deatrichMar 24 2011
Resetting RSA/IMM interfaces on IBM server nodes ............deatrichJan 26 2010
Sun-Oracle
Getting a console on Sun Blades .............................deatrichJul 28 2010
Getting a console via ssh to ILOM interface .................deatrichApr 12 2017
Getting a serial console via ssh to Sun Chassis bl ..........deatrichJun 20 2017
Requesting a signed cert. for Sun remote interface ..........rajandevJun 22 2016
Resetting the ILOM Service Processor .......................vitaliykJul 15 2010
Sun ILOM Configuration ......................................nicoleDec 07 2009
Performance
Switches
S4048-ON- OS9 upgrade procedure .............................vitaliykNov 24 2022
The OS9 upgrade procedure for Dell S3048-ON type s ..........vitaliykNov 23 2022
Infrastructure
Miscellaneous
10zig Terminal Set-up .......................................deatrichJan 06 2015
Monitoring
Atlas UPS Battery capacity ..................................vitaliykJun 09 2010
Block snmp trap client IP to access trap server .............vitaliykAug 05 2010
Changing nagios configuration files .........................deatrichJun 17 2009
How to create downtime during Lathams PM on nagios ..........vitaliykNov 03 2015
Identification of power interruption with Nagios s ..........vitaliykMar 11 2022
Migrating ganglia rrds to a tmpfs file system ...............cpayneJun 15 2009
Monitoring continuity through node recycling ................cpayneDec 05 2008
nagios powerdown workers setup ..............................cpayneAug 05 2009
Pager replacement notes .....................................deatrichJan 28 2011
Passphrase process on admin nodes ...........................deatrichAug 02 2012
Removing mega-spikes from ganglia plots .....................deatrichMar 01 2012
Scheduling downtimes in Nagios for groups ...................cpayneJan 07 2009
sendpage setup ..............................................cpayneAug 13 2009
Updating Oracle RPMs on nagios hosts ........................deatrichAug 05 2009
Worker node monitoring ......................................deatrichJul 24 2012
Alarms Email
Alarms setup on lcgmx .......................................deatrichSep 29 2008
Refreshing ca-bundle.crt ....................................deatrichJun 27 2011
Renewing test signed email ..................................deatrichMar 10 2011
Send a signed message using openssl .........................vitaliykSep 04 2019
Cron Jobs
Network
Access data center through IPs provided by SFU IT ...........dqingMar 15 2022
Accessing web interfaces of devices on the private ..........cpayneFeb 20 2009
Changing a node hostname and/or IP address ..................cpayneDec 05 2008
Configure aggregation device on Solaris .....................dqingOct 29 2009
Configure Linux Virtual Local Area Network (VLAN) ...........dqingOct 02 2009
Creating a generic squid for http/https traffic .............deatrichAug 24 2018
Example openvpn setup on workers ............................deatrichJun 23 2015
iptables count-rule .........................................simonApr 17 2009
Modifying DHCP for t1dev devices ............................deatrichMay 05 2014
Renewing proxy for nagios monitoring ........................deatrichAug 12 2009
DNS
DNS slaver server setup on NAT boxes ........................cpayneMay 15 2009
Master / Slave setup in Tier-1 ..............................deatrichMar 07 2017
Modifying T1 DNS on NAT boxes ...............................deatrichJan 20 2012
Modifying t1dev and lcg DNS on t1-nsX naming serve ..........deatrichApr 28 2014
Updating DNS host info via Ansible ..........................deatrichMar 08 2017
IPv6
Firewall for IPv6: ip6tables ................................deatrichOct 28 2016
IPv6 configuration files for networking .....................deatrichOct 28 2016
Tuning
Solaris
SunOS 5.10 ..................................................vitaliykFeb 22 2010
PPS
Dell Blade Usage ............................................deatrichSep 28 2012
ROC
ROC - virtual machines ......................................aywongOct 27 2015
ROC Nagios instance installation and configuration ..........dqingSep 28 2010
ROC Nagios instance operations ..............................dqingSep 28 2010
ROC nodes and their relations ...............................dqingSep 28 2010
ROC WMS plus MyProxy installation and configurati ..........dqingSep 29 2010
Security
Changing login passwords ....................................deatrichJan 14 2014
Grid User Security Notification Handling ....................deatrichMar 31 2009
Looking for connections from evil hosts .....................deatrichDec 10 2010
Monitoring for suspicious SSH connections-authmond ..........samlaicsJul 16 2014
Services and Ports on Servers ...............................deatrichMar 17 2014
Un-banning an IP address under fail2ban service .............deatrichJun 04 2019
System Admin
Admin Services
Managing the HOLD queue on lcgmx ............................deatrichSep 26 2013
Spamassassin set-up on lcgmx ................................deatrichSep 26 2013
Ansible
Introduction to Ansible .....................................deatrichSep 17 2015
Managing /etc/hosts and the pdsh variables configu ..........deatrichJan 20 2016
Set up your own ansible devel area on host ansible ..........deatrichMar 12 2019
Set up your own ansible test area on ansible ................deatrichJan 15 2016
yum playbooks ...............................................deatrichJan 28 2016
Roles
Role autofs certified .......................................deatrichApr 20 2016
Role base certifed ..........................................deatrichApr 25 2016
Role cvmfs certified ........................................deatrichApr 25 2016
Role dns-client certified ...................................deatrichApr 28 2016
Role fetch-crl certified ....................................deatrichApr 28 2016
Role ganglia certified ......................................deatrichApr 28 2016
Role generic-node certified .................................deatrichApr 22 2016
Role grid-certs certified ...................................deatrichApr 20 2016
Role ipmi certified .........................................deatrichApr 28 2016
Backups
Adding new nodes to Dirvish .................................deatrichJun 16 2014
Back up a host before reinstalling it .......................deatrichApr 08 2015
Restoring Files .............................................deatrichSep 29 2008
Weekly Backups
dCache copies of the weekly backups .........................deatrichSep 29 2008
How the backups are done ....................................deatrichSep 29 2008
Offline backups .............................................deatrichSep 25 2019
Decommissioning
Server decommissioning checklist ............................deatrichMar 08 2018
Unconfigure a server for removal ............................deatrichMar 08 2018
Filesystems, LVM, RAID
Growing an LVM Volume .......................................deatrichOct 22 2008
Limit the devices scanned/used by LVM .......................deatrichJun 03 2010
Typical NFS configuration in TRIUMF Tier-1 ..................deatrichFeb 09 2009
cvmfs
Setting up cvmfs on Workers .................................deatrichJun 20 2011
Installation
Creating an installation/recovery USB drive .................deatrichJan 25 2011
Installation Infrastructure .................................deatrichFeb 02 2009
Installing an apache.org tarball ............................deatrichMay 06 2019
Rescue a server via PXEboot .................................deatrichFeb 20 2015
Summary of Solaris to Linux process on x4540 ................deatrichApr 23 2015
Jumpstart
Kickstart
Anatomy of a Tier-1 kickstart file ..........................deatrichFeb 09 2009
Modifying the pxe-boot initrd image - SL 5 ..................deatrichFeb 02 2012
Modifying the pxe-boot initrd image - SL 6 ..................deatrichJun 25 2013
Modifying the pxe-boot initrd image - SL 7 ..................deatrichMar 29 2016
Quattor AII
Protecting the Quattor web tree .............................deatrichFeb 09 2009
Quattor installation of a worker or pool node ...............deatrichOct 07 2008
Software Mirror
Adding a new SL distribution to the mirror ..................deatrichApr 27 2015
Mirror Structure and Maintenance ............................deatrichOct 06 2008
Maintenance
Move sudo messages to a separate log file ...................deatrichAug 07 2009
Setting up temporary log file rotation ......................deatrichOct 15 2012
Patching and Upgrades
Applying a Solaris patch bundle .............................deatrichDec 06 2010
Preparing dCache Solaris packages ...........................deatrichOct 04 2011
Preparing java 1.6 Solaris packages .........................deatrichFeb 10 2011
Preparing java 1.7 Solaris packages .........................deatrichMar 11 2013
rpm db issues ...............................................deatrichJul 19 2012
Updating java6 on Solaris ...................................deatrichDec 13 2010
Updating java7 on Solaris ...................................deatrichMar 11 2013
Updating lcg-CA RPMs ........................................deatrichOct 06 2008
Updating panda-server user interface library ................deatrichSep 26 2016
Usage of dracut command and initramfs .......................dqingNov 13 2020
RPM Notes
build a set of postrgresql RPMs with a different r ..........deatrichNov 19 2008
Building jpackage-style java RPMs ...........................deatrichMay 09 2009
Solaris Notes
User Support
Dial-in information for Tuesday meetings ....................deatrichDec 10 2013
External access to SFU-WTB ..................................deatrichOct 31 2018
How to interpret rrd info ...................................apetkauJul 07 2010
How to use rrd conversion tools .............................apetkauAug 27 2010
How to use the gridusers script .............................apetkauAug 20 2010
How to use the ipsearch script ..............................apetkauAug 20 2010
Pointers for new Tier-1 Staff ...............................deatrichSep 29 2008
Setting up a host to generate Globus grid certific ..........deatrichApr 29 2009
Setting up FreeNX service on your desktop ...................deatrichMay 28 2009
Solaris / Linux user issues .................................deatrichSep 30 2009
Tunneling nxclient via Westgrid .............................deatrichJun 24 2014
User setup pointers on Tier-1 Dell Mini Inspiron ............cpayneJan 07 2009
Using our local firefox/java tarball ........................deatrichMay 29 2009
Build Environment
Basic General Git Usage .....................................samlaicsJul 28 2015
Building RPMs as a non-privileged user ......................deatrichNov 19 2008
Introduction to Build Environment ...........................deatrichAug 11 2015
Introduction to git for Tier-1 users ........................deatrichAug 11 2015
New user setup on build host ................................samlaicsMay 29 2015
Populating the mirror with RPMs .............................samlaicsJul 31 2015
Using CVS from a shared account .............................deatrichJun 30 2010
Using CVS in the Tier-1 Centre ..............................deatrichJun 28 2010
Virtualization
Docker
Docker client commands for managing containers ..............deatrichSep 20 2016
Docker client commands for managing images ..................deatrichSep 20 2016
Docker terminology ..........................................deatrichSep 20 2016
KVM
Balancing clients on gridvm1/gridvm2 ........................deatrichDec 17 2013
Creating a KVM guest using iSCSI LUNs .......................deatrichJan 11 2013
Creating a KVM guest using LVM ..............................deatrichJan 21 2011
Getting a console on a KVM guest ............................deatrichDec 17 2013
Hot-plugging disks on a KVM guest ...........................deatrichJun 28 2016
Installing a KVM server with bridged guest network ..........deatrichJan 21 2011
Live migration of a KVM client ..............................deatrichDec 17 2013
Live-migrating all clients ..................................deatrichDec 17 2013
Setting up LVM-based storage pools ..........................deatrichMar 20 2015
RHEV
Bringing RHEV up after a shutdown ...........................deatrichJan 09 2019
Complete RHEV maintenance procedure .........................deatrichSep 25 2018
Create a new virtual machine ................................deatrichMar 06 2019
Setting up SNMP traps on RHEV ...............................deatrichOct 06 2018
Updating RHEV at minor releases .............................deatrichMar 21 2019
VMWare
Xen
Creating a new paravirtual Xen guest on lcgvm* ..............deatrichSep 24 2008
getting information about Xen clients .......................deatrichOct 06 2008
Growing diskspace on a Xen guest ............................deatrichOct 27 2008
Hot-plugging disk devices on a Xen guest ....................deatrichOct 27 2008
Installing a Xen server with bridged guest network ..........deatrichMay 08 2009
Reducing diskspace on a Xen guest ...........................deatrichSep 30 2010
Save a Xen guest ............................................deatrichJul 31 2009
Switching host/standby VM guest pairs .......................deatrichMay 25 2009
Web
'build' web page generator ..................................samlaicsJul 20 2015
Panglia setup ...............................................deatrichNov 15 2013
Setting up Let's Encrypt ....................................deatrichSep 13 2018
Some help using Recipes Web site ............................deatrichSep 30 2008
Synchronizing rrd data between panglia servers ..............deatrichNov 15 2013
Updating the 24x7 guide .....................................deatrichOct 29 2010
Updating the Tier-1 Notification Document ...................deatrichOct 15 2013
XML Script Status Pages .....................................samlaicsJul 15 2014
Remote Access