FAQ: Experiment Installation & Maintenance​

Question:
What are the steps necessay to install my experiment in the tunnel?

Answer:
Assuming your experiment has been approved, Spencer Gessner will organise the installation.

1. Get your apparatus to SLAC.

2. Stage your apparatus outside of FACET (we have space available in End Station B's clean room and we can also arrange space in MFD's *very* clean room) to confirm that all the neccessary parts are present and in good condition to be installed. Because of this stage, you need to have your parts delivered to SLAC well in advance. At least a day is necessary to stage and check all the items and then if there are any issues, it could take around a week to resolve.

There fore you must deliver your apparatus to SLAC 1-2 weeks before installation. Sooner is preferred.

3. Install it. Usually the experimenters are aided by Test Facilities staff or MFD Vacuum technicians depending on the type of work involved. Spencer arranges the support. Usually this will occur during FACET downtime.

Sometimes we install apparatus on access days though this is not preferred and due to the difficulties in scheduling this, this can delay beam delivery to the experiment.

For a description of how to submit work in FACET during a downtime, go to the next question.

For a description on how to submit work in FACET for an access (called "PAMM") day, please find the question below.


Question:
​How do I write the procedure for experiment installation?

Answer:
Contact the PAMM coordinator Ivan Rajkovic and tell him what you would like to do at least two weeks in advance (but preferably even earlier) for efficient planning of the activity. The sooner the better in order that we can guarantee arrange appropriate support. Some notes on this process can be found here: https://confluence.slac.stanford.edu/display/FACET/PAMM+Planning 

Note that Spencer needs to have a written procedure from you.
 
The procedure should include the following details:

  • Purpose/goal for the work
  • Number of people involved (give names if known)
  • Steps for the work (with time duration estimates per step and in total)
  • Hazards involved and safety measures that must be taken

The level of detail for the work should increase with its complexity and whether the job is "new" rather than routinely performed. Routine and simple jobs may only be briefly described.

There is a specific template for laser work and this laser work is expected to be discussed in a laser meeting held by the SLSO. Contact Brendan O'Shea for an invite to this and a link to the template. This is because the laser work is highly complex as so many experiments use the laser and the impact from your work could be felt by them. The template calls for photos of the place where the work will happen and this is really encouraged as it helps the work planning and discussion a lot! So plan to go down to the tunnel and take photos about a month before you want to do the work.
 
Examples of good procedures can be found here and here. These are written by two very different user groups in very different styles but both show good planning. This procedure uses the laser template: https://docs.google.com/document/d/1Kh7iW6_8dbKeAAKRg-F8Qvd1xazLeatbmmDG...
 
The main purpose of this procedure is for you, the user. It should be written in the format you find easiest to work to. Writing it should be a good exercise for you that ensures safe and successful tunnel work.
 
A generic safety analysis for experiments in FACET can be found here and here. This can be useful to consult when you plan your work though please do not copy and paste hazards/mitigations - only write the key points that actually pertain to your work and are useful in the execution of it. 

Question:
​What are the rules for in-vacuum hardware?

Answer:
Components need to be vacuum compatible to 1e-6 Torr (or mbar).

​General guidelines for bolts in vacuum chambers (written by Robert Ariniello):

  • If the bolt is threaded into a blind hole (anything that isn't a through hole) it needs to be vented. Otherwise a small pocket of gas can be trapped under the bolt and leak out slowly over time.
  • If the bolt is threaded into a stainless steel part, it should be coated. If it isn't coated galling can occur and the bolt can seize in the threads. This isn't strictly required for a good vacuum, but if bolts aren't coated they may be very difficult to remove. Silver plated is the most common coating but you can also use MOS2 or gold.
  • Bolts must be made of stainless steel and not have zinc coatings.

Silver plated bolts can be purchased from McMaster-Carr and UC Components, vented bolts can be purchased McMaster-Carr, UC Components, and Thorlabs, bolts that are both vented and silver plated come from UC Components. It took around 1 month to get bolts from UC Components in November when I last ordered from them.

Some of the holes on the top of a UTS stage are blind and need to be vented. The PB [Picnic Basket vacuum chamber] breadboard is stainless steel and needs to have silver bolts, but they don't need to be vented.

Question:
What is a PAMM?

Answer:
A PAMM is a "Planned Access for Maintenance and Modifications". Typically during FACET running there was a PAMM every one to two weeks on Tuesday and Wednesday. This is subject to change for FACET-II as we find our feet.

The beam is shut off at a pre-arranged time. After a one hour radiation cooling off period, the tunnel is ready for access by Radiation Physics Field Operations (RPFO). RPFO survey the beamline. After the survey, RPFO de-briefs the access coordinator and any workers present for the start of the access.  

Only jobs that have beed pre-approved and released are allowed to be done.

If you keep the PAMM Coordinator (Ivan Rajkovic) aware of your access needs in general, we can coordinate the work.

 

Question:
How are the jobs for a PAMM requested and released?

Answer:
​1). Requested work procedures are sent to the PAMM coordinator (Ivan Rajkovic). Usually this has to happen two weeks before the PAMM.

2). Ivan puts them into CATER (Comprehensive Accelerator Tool for Enhancing Reliability!).

3). Area Manager, Emad Hamrah, reviews the CATER and releases the job (or not if he finds fault with the job).

4). Released and unreleased jobs are reviewed in a PAMM planning meeting that is held across the entire lab. Ivan Rajkovic represents all experimenter jobs though attendance is open to all. The aim of this is to ensure that there is no conflicting work e.g. electrical or controls outages.

 

Workers are *required* to have authorization from their supervisors prior to work. Non-employees ("users") are usually supervised by Ivan Rajkovic who authorises the work by reviewing the procedure and assigning training appropriately. If there is any doubt, Ivan will contact the supervisors to check whether the worker is authorised to perform the requested job.

If the term "release" in this answer are unfamiliar to you, you shouldn't be requesting jobs in FACET! All these terms are explained in course 120 which is required for working in FACET. You can retake the course at any time as a refresher or read the ES&H chapter on Work Planning and Control.

 

Question:
How can I add an item to the list of jobs for a PAMM?

Answer:
As soon as you know you want to work in the tunnel, contact the PAMM coordinator, Ivan Rajkovic, and explain what you would like to do about two weeks if not more before the upcoming access. Ivan needs to know approximate duration and some details of the work especially location and number of people. This puts the work on the schedule. Ivan will let you know the date the work is scheduled. Good ways to contact Ivan are through slack/email or send an Airtable PAMM job request.

Send Ivan the procedure for the work (or provide one with your Airtable PAMM job request).

The procedure should include the following details:

  • Purpose/goal for the work
  • Number of people involved (give names if known)
  • Steps for the work (with time duration estimates per step and in total)
  • Hazards involved and safety measures that must be taken

The level of detail for the work should increase with its complexity and whether the job is "new" rather than routinely performed. Routine and simple jobs may only be briefly described.
 
Examples of good procedures can be found here and here. These are written by two very different user groups in very different styles but both show good planning.

Due to the incredible complexity of the laser optics, laser work needs a lot of coordination between user groups and should always be discussed and okayed in a laser meeting with other user groups present prior to execution. Contact the SLSO Brendan O'Shea to get an invite to the laser meeting. There is also a particular format for the procedure in this case that emphasizes such things as taking photos of the planned locations - this is proving essential for people to discuss the work and understand how it impacts the multiple experiments that share the laser.
 
Note that however the main purpose of this procedure is for you, the user. It should be written in the format you find easiest to work to. Writing it should be a good exercise for you that ensures safe and successful tunnel work.
 
A generic safety analysis for experiments in FACET can be found here. and there is a Job Safety Analysis document here. This can be useful to consult when you plan your work though please do not copy and paste hazards/mitigations - only write the key points that actually pertain to your work and are useful in the execution of it. All workers should understand the hazards of their own hardware.

User groups who routinely write procedures are more successful on their PAMM days. Experienced user groups can attest to this!

 

Question:
What is the deadline to add to the list of approved PAMM jobs?

Answer:
​It's really hard to plan PAMM jobs if they are not inputted as early as possible but we do of course understand that some work comes at the last minute e.g. if something breaks. We will always be open to such last minute jobs if they have a legit reason. However, we really do want the majority of jobs to be on our radar for planning purposes two weeks in advance. Effectively, it becomes really challenging to add new jobs without a reason after the Thursday prior to the access. All PAMM jobs need to be reviewed prior to release so if you miss this deadline without a good excuse, we are unlikely to be able to review it in time for it to proceed.
 

Question:
Why are you asking for the PAMM jobs so early? (I don't know what I want to do two weeks in advance!)

Answer:
The idea is to encourage planning ahead and prevent last minute additions. See the "swim lanes" image on this site to understand the process a little better and to see why it takes time.

 

Question: ​
​What training do I need to work in the FACET tunnel?

Answer:
​See the FAQ on Training.

 

Question:
​What time of day will I do my work?

Answer:
There are usually multiple groups doing multiple jobs in the FACET tunnel. Without staggering jobs, the working area can get congested. This does not lead to efficient or indeed safe work. Therefore, we try to coordinate the jobs to different times of the day. There are also hazards involved with some of the jobs that we don't wish to expose people to unnecessarily so these jobs are scheduled quite strictly and the number of people in the tunnel controlled.

As a result, it is hard to tell until all jobs are submitted the exact schedule for the day. It may not be known until the Monday prior to the access exactly what the script is though every attempt will be made to have a broad plan on the Friday prior to the access.

We may request that people work on an owl shift (midnight - 8am) in some circumstances.

Example PAMM schedule here.

 

Question:
​I just want to go down for a look- what do I need to do?​

Answer:
​If you are just looking and not performing any work, you just need training. If you are current in AD103 (FACET Tunnel Hazards/ on-site orientiation) and course 116 (Radiation Worker) then you don't need specific work release. However, it is best to tell the PAMM coordinator of your intent so email/call/add to the google spreadsheet or otherwise communicate this to the coordinator. The coordinator will then ensure that you are informed of the work schedule, possible hazards and PPE requirements. This will save time so you don't go to FACET and then realise that the work that is going on precludes your visit.

Question:
What are the different FACET tunnel access modes?

Answer:
There are two FACET tunnel access modes (Permitted & Guarded. There is no controlled access in FACET):

1). Guarded Access with Radiation Physics Escort - this is designed for short (less than 30 minutes) accesses for one or two experimenters. This access mode minimizes the beam time lost because by guarding the entrance there is no need for a search after the access.

2).  Permitted Access - this is for when longer access is needed or more thanone or two experimenters need to go into the tunnel. This access mobe is used for PAMM. If such an access is required during run time, the experiment may lose as much as a shift of beam so the decision to go in is not done lightly.

More information on the these access modes, how they are done and the impact is found here.

 

Question:
​How do I make an access during a shift?

Answer:
This constitutes a change in program.

If your program changes, you must contact Mark Hogan immediately (and/or your experiment POC.

Once approved, follow the steps below.

​1. Plan it. What do you want to do? Make sure you know step by step and have all of the parts to hand. Put the plan in the elog.

2. Estimate time. How long do you require?

->  If it is <30 minutes, it can be done under guarded acces. Otherwise, it needs to be done under permitted access.

3. Tell the EOIC that you need an access. Tell them what time you want the access, how long it will last and what type of access you want. The EOIC may advise you that other times will be easier to support and you may enter negotiations!

4. Ask the EOIC to call RPFO so they can survey. (RPFO need at minimum 2 hours warning).

5. Remember to turn the beam off an hour before you need to go in!

6. Go in. Access procedure is here.

7. Whilst in access, it is very important to let ACR know what is going on. Otherwise your guard will leave when time is up and we will go into permitted access and a search will be necessary. This will cause many hours of delay. Best thing to do is to have someone in the trailer to field calls between the tunnel and ACR. No one except this dedicated person should call the tunnel for "updates" because this can delay work in the tunnel.

 

 

Question:
​What do I do if something isn't as it should be in FACET tunnel?

Answer:
​For example, lights are flashing or there is water on the floor or something appears broken.

Call ACR x2151 and report this.

Depending on the issue, you may also want to tell the Area Manager Emad Hamrah, the PAMM Coordinator or your Experiment POC. ACR operators can help you decide whether more people need to be informed and usually make the phone calls.

 

Question:
​How do I access FACET during downtime?

Answer:
When FACET is in downtime it is usually but not necessarily in permitted access. We may lock FACET for reasons such as electrical outages, safety system testing or for certification purposes. In addition, there may be activities going on inside FACET that are incompatible with general access such as electrical work or class 4 laser work.

Please see information below about entering and working in FACET during the downtime assuming FACET is in permitted access.

Do I have to be RWT1 or GERT?
--------------------------------------
FACET is usually a Radiologically Controlled Area (RCA). You therefore need to have GERT (up-to-date!) to enter. However be on the watch for changes in postings in case it is a Radiation Area. If FACET is a Radiation Area, you need to be a Rad Worker for access and you need to sign the area RWP.

However note: even if only GERT is needed to access FACET, Radiation Worker is needed to touch anything within the housing that was there when beam was on! Any GERTs that access FACET must keep hands-in-pockets. Even when FACET is a RCA, you need to be RWT trained to do any work. GERTs can enter FACET but they cannot handle any material.

 

Remember that anything taken out needs to get surveyed.

 

Is Area Manager release needed or not?
-----------------------------------------------

Green work is released in FACET if you...

– Always follow the AHA (displayed at the entrance),
– Remember your training (you should have course AD103 or be escorted by someone that has),
– Observe ongoing activities in the area in the event hazards may be present which require your attention.

You do not need further release from the Area Manger if this is followed. If you do not have AD103 training, you need Area Manager release.

 

Yellow work (non-observational work/work with tools):

Work with the PAMM coordinator if you have any non-green work to perform. Work that is done in FACET requires Area Manager approval. This is done in the same manner as with a PAMM - indeed, the PAMM planning spreadsheet is still used for downtime work.

Tours:

Large groups of visitors, tour groups, photo opportunities and equivalent activities require a release from the Area Manager.

References:
Work Planning and Control training (course 120):
https://internal.slac.stanford.edu/wpc/

Access Requirements (GERT 115 and RWT1 116 training):
http://www-group.slac.stanford.edu/esh/eshmanual/references/radReqWorkEntry.pdf​

Question:
What type of motor should I use?​

Answer:
A stepper motor.
 
DC motors always have some sort of encoder or tachometer that feeds back  position information to a controller that commands the DC motors position. Encoders all contain semiconductors of one sort or another and any semiconductors that have been place close to the beam have always failed within  a couple of weeks.  A tachometer that  uses magnetic elements to produce pulses could work- a Hall sensor would be trouble.
 
Stepper motors on the other hand are open loop and rely on precision manufacturing to obtain their accuracy per step and have no feedback for position. Hybrid stepper motors also contain magnets but at this point magnetic failure due to radiation has not been a problem.  In a high radiation environment, if your experiment can work with the stepper motor error specifications it is best to go with the stepper.
​

Question:
What cameras are at FACET​?

Answer:
Currently FACET fully supports:

http://www.alliedvisiontec.com/us/products/cameras/gigabit-ethernet/manta/g-095c.html

 Very likely all Mantas and Prosilicas listed in the website are supported by areaDetector. Besides the Manta 095B we use commonly at FACET, we are using the Manta G146 and the Prosilica GX1050 in LCLS.
 
There are cameras from other vendors also supported by areaDetector. They are listed here: http://cars9.uchicago.edu/software/epics/areaDetector.html
 
More details about areaDetector: http://cars9.uchicago.edu/software/epics/areaDetectorDoc.html