FAQ: Experiment Running

Question:
How can I print to the FACET e-Log?

Answer:
​While not clear from the help in the FACET e-log, it is possible to print directly to the log. This may save time by allowing a user to simply print from their machine instead of logging in and uploading. After printing, the e-log entry will need to be updated with a title, author, etc.

The attached pdf document explains how.

Question:
​What is the FACET e-log and why do I need to know about it?

Answer:
The FACET e-log is an electronic log book that contains details of the FACET run. It used both by the FACET commissioning team and by the experimenters to record running details.

Question:
What do I need to put in the elog?

Answer:
​It is very much appreciated if all shifts are well documented with entries throughout explaining what you are doing and what is happening with the machine. The elog is the primary source of communication and it saves people from having to phone you to work out what you are doing and whether there are any problems.

You should document work that you did in the tunnel after a PAMM or user access.

Question:
What is the elog etiquette?

Answer:
​ALWAYS put in names under the author field. Often when people print to the elog, they forget to edit the entry to put their names in rather than printmeta.

Explain what you are doing rather than post a series of pictures without any comments. If you change a setting, note that you changed it rather than just show the result without explaining what you did to achieve it.

You can edit entries but be careful when you do so not to remove actual content. If you write something that you later think wasn't true, don't delete it. That is not good science. You can edit the entry to include an additional comment.
 
Question:
What counts as beam delivery/downtime etc?

Answer:
​Delivered
Beam is available for the scheduled program.

User Off
Scheduled FACET user is not able to take the deliverable beam or makes an access.
 
Tuning
Beam conditions do not meet the requirements to carry out the scheduled program, and accelerator adjustments are being made to bring the beam to a deliverable state.
 
Config Δ’s
User has requested a configuration change. Time is the time to restore a deliverable beam.
 
Accel Down
Accelerator hardware or software problem is preventing the beam from being delivered. User hardware down is NOT Accelerator Down time.  Include specific problems causing downtime in the shift summary, with approximate time (-x.x hours). The EOIC (or their designee) is responsible for creating a CATER(s) to document all accelerator downtime for the shift.
 
Scheduled Off
PAMM and POMM activities and BCS Checks.

Question:
What is the template for a shift summary?​

Answer:
At the end of your beam time, the experimental team on shift must put in a summary. This really helps communication with the users, accelerator physicists and accelerator operators. We also use the reports from you to write the reports to DOE about how we are using beam time.
 
To generate and summary go to http://ad-ops.slac.stanford.edu/facet-shift-report/shift_summary_form/ and fill out the form. You can edit your report as many times as you like and save it to the database. When you are completely finished click on "Save and Submit to Logbook" and the report will be saved in a data-base and also an entry will be made in the FACET e-log. Please check the e-log to ensure the shift summary was added. The link at the bottom of the database report should take you to the e-log entry. If it does not please look at the e-log corersponding to the "Time to Post to E-Log".
 
ONCE THE REPORT IS SUBMITTED YOU WILL NOT BE ABLE TO CHANGE IT. But there is an admin mode. Contact your Experiment POC for help if needed.
 
Time and date format are YYY-MM-DD HH:MM:SS. You have to follow this format.
 
Sometime Internet Explorer does not behave. Google Chrome will work under these circumstaces.
  
 
*******OBSOLETE INSTRUCTIONS/USE IF ABOVE DOES NOT WORK********
 
The facet elog uses a mark-up text so copy and paste the template below. The mark-up (exclamation marks) make the subsequent text into headings.
 
Parasitic programs also need to put in their own summary of beam time. Mark it as "parasitic" so it's clear it wasn't the main program.
 
An example of the perfect summary. (Needs to be on SLAC network to view- VPN if off-site or on visitor wireless)
 
Template for shift summary:
 
!Program
Main experiment:
Shift:
Shift leader/shift personnel:
!Goals
!Progress
!Problems
!To do on next shift
!Brief Summary (~2 sentences that are copied into the DOE report - see examples)
!Delivery to main program
 
Useful beam time: X hours
Unscheduled down time of accelerator: X hours
Time when beam was good but experimental equipment failed (aka "user off"): X hours
% time accelerator physicist available: ​

Question:
​Why are you asking for this information in the shift summary?

Answer:
We need to report back to various people- other users, accelerator physicists, accelerator operators and, of course, DOE.

The DOE reports lift a few things from the shift reports- key amongst these is the summary of the beam time.

You can see what other people have written as their summaries here.

Question:
​How do I write the perfect "brief summary"?

Answer:
The "brief summary" in the report at the end of your shift is sent to the Department of Energy.

It should not contain terms that only the experimenters or people at the facility know. General accelerator terms are okay.

It should be succinct. Two or three sentences. It should give a sense of the progress towards meeting your goals.

Examples (courtesy of E200, E210 and E201 respectively):

1.
We have identified the best strategy to perform a tomographic reconstruction of the longitudinal phase space in a time-efficient manner, allowing to considerably reduce the sensitivity of the measure to the drift of the transverse deflecting cavity and of the machine.

2.
Goals for the shift were achieved. Machine response matrix was identified and a reduction of eroneous dispersion by a factor 4 was obtained for the first half of the linac.

3.
Measured ~540MeV/m gradients in a 10cm long, 450um inner diameter dielectric tube with a 25um copper coating. The 54MeV energy change is a world record for dielectric wakefield accelerators. The spectrum of the wakefields excited in the structure agreed with the design predictions.

Question:
​What is the current shift schedule?

Answer:
Current schedule is posted here: https://facet-ii.slac.stanford.edu/user-resources

If you have questions or requests for changes or additions, contact your Experiment POC.

Question:
​How many shifts will my experiment get?

Answer:
This is usually arranged to a rough approximation in advance of publishing the schedule through communication with the Experiment POC. They will request and digest your beam requests and work with you to develop a reasonable number of shifts to fulfil science goals.

If your results during a run indicate a need for further shifts, a detailed proposal on how many shifts, the goals and the analysis of shifts so far should be produced. This should be sent to the experiment Point of Contact.

Many experiments at FACET-II get single shifts sporadically across a run (gaps between shifts can allow for data analyses and are usually part of a user beam request). Long programmatic studies can expect to get a series of shifts across a week or two for example. The assignment of shifts usually comes down to the science need and also the recommendations of the Program Advisory committee (PAC) for priorities for the FACET-II facility.

Question:
​Who is the point of contact for my experiment?

Answer:
Invited experiments are each assigned a point of contact. If you don't see your experiment on the list and it was invited for beamtime, please contact one of the Experiment POCs for guidance. They likely just need to update the webpage. SLAC proposals only have a SLAC POC who is typically also the PI.

Below are roles and responsibilities of the external and SLAC POC. 

In general:

The External POC (if there is one) is expected to drive the schedule for the experiments.

The SLAC POC is facilitating in the field.

The External POC…

• Ensures that installation work is on the schedule for the access periods that they need to support beam time.
• Ensure that beam time procedures are submitted for review to FACET-II Director (Mark), Operations head (Jerry), laser safety officer (Brendan) and User Manager (Christine) at least 2 weeks before anticipated beam time.
• Ensures that there are experimenters available for installation and beam time
• Ensures that experimenters are adequately trained on the experiment hardware

(Note: In cases where there is only a SLAC POC, the SLAC POC does this role also)

The SLAC POC…

• Assists in the development of beam time procedures (provides feedback, suggests improvements and lessons learned from other experiments)
• Works with Operations head Jerry to assign a time for the beam and communicates that to the external POC
• Identifies and coordinates facility support needed (note: facility support does not involve performing the experiment)
• Is on call during the beam time in case the experimenters have questions or need to change the procedure. Changes to the procedure may need to be re-reviewed if they deviated from the established envelope of approval and the SLAC POC is responsible to assess if this goes beyond the established envelop of approval.

Question:

​Why are you asking for shift plans and procedures?

Answer:
​The total number of experimental shifts is relatively low at FACET-II. Efficient planning for experimental time is therefore required for the experiment to be successful.

The facility needs to be provided with a detailed, step-by-step plan for your beam time so we can be certain that the planning has been completed. This also means that we can plan better on our end to provide good beam and support.

Question:
​Can we be guaranteed good beam?

Answer:
At FACET-II we focus on good quality beam delivery.

The beam will be fuly characterised and tuned before being delivered to experiments.

This may result in a delay to delivering beam to the experiment if more work is needed to get good beam. Your patience when your shift does not start exactly on time is appreciated!

Please do not ask for better quality than you need. This will delay your beamtime as the accelerator team work on achieving the requested beam parameters. Equally please do not accept the beam unless your needs have been met.

Question:
How long is a shift?

Answer:
In order to deliver good quality beam, FACET-II spends a lot of machine time to daily maintenance activities and machine development. We therefore do not schedule experiments around the clock (maximum shift length is usually 12 hours).

Shifts are of flexible duration according to the science needs. Users must have a written procedure for their work that clearly indicates the time that is needed. That procedure will be agreed upon with FACET staff and then executed. If the tasks take longer than expected and the agreed upon length of the shift needs to be extended, this can usually be accommodated but will require discussion with facility staff.

 

Question:
What do I need to do to get my beam time approved? (What is the timeline for what is due to be submitted when?)

Answer:
Pre-shift requirements at FACET-II are:

1) a procedure for the beam time and

2) analyses of previous shifts.

This requirement is for the benefit of our user groups. Good planning leads to better results!

Users must have a written procedure for their beam time that clearly indicates the activity, requirements from the beam and operators and the time that is needed. Guidelines are given (see "how do I write a good procedure" in the FAQ). 

The Points of Contact should work together to develop the procedure for the beam time.

Timeline

Before the run gets scheduled

Coarse outlines including goals, expected analyses and beam and hardware requirements are expected in advance of the run so we can appropriately schedule the experiment. Please send these to the FACET-II User Manager. The step by step procedure is not required at this point but the sooner it is thought about, the better prepared you will be (it may be that only in thinking step by step, you will realize a missing piece of hardware or forgotten beam requirement).

Two weeks before anticipated beam time

The procedure for your beam time studies needs to be sent to and reviewed by representatives from the Test Facilities department, the Beam operations department and the advanced accelerator research department. This is specifically Christine Clarke, Jerry Yocky and Mark Hogan though during periods of vacation or other absenses, there will be someone else from their department. If there is use of the sector 20 laser, the procedure additionally gets reviewed by the System Laser Safety Officer Brendan O'Shea. This needs to be done two weeks before the anticipated beam time.

The Points of Contact will work together to coordinate support for the experiment both from the user team or from the facility as needed. The Points of Contact will work with the Beam Operations group to schedule beam time for when beam conditions required and the support needed is available. 

The Points of Contact are responsible to communicate the schedule to the experimenters and ensure there is adequate staffing for the shift.

It should be noted that as the SLAC-based POC is supporting multiple experiments and improvement projects, the external POC is expected to drive this process.

Day of beam time

Jerry Yocky or designee instructs the accelerator operators on shift on how to set up the beam and check out the hardware as requested in the procedure.

Start of Shift

A Point of Contact or designee will then coordinate with ACR - check in with the ACR and operator that is driving for FACET-II at the start of the shift. Introduce yourself, ask them about the state of the machine and then take a few minutes to give them the elevator pitch about what you’re doing, why it’s cool. Then cover practical things like the how - puff the gas jets, ionize it with a laser and look at effect on the beam using downstream profile monitors or whatever. Especially important for them will be to know if you have expectations about changing the state of the accelerator (charge, compression, waist location, IP betas, spectrometer…). It’s important that they understand the recipe, ranges and have blessing to do so from Jerry. This kind of thing is better clarified at 4PM than 4AM. One of the things we would do at FACET is make an elog entry that has a link to and high level summary of the approved procedure and POC with phone. Examples below:

Shorter:
http://physics-elog.slac.stanford.edu/facetelog/show.jsp?dir=/2015/46/13...

Longer:
http://physics-elog.slac.stanford.edu/facetelog/show.jsp?dir=/2015/46/14...

There is no exact template. Anything will be better than nothing.

During the Shift

If the procedure changes during the course of the beam time, beyond the approved envelop of activities these deviations should be checked with FACET-II staff (FACET user manager, division director etc.). This includes if tasks take longer than expected and the agreed upon length of the shift needs to be extended- this can usually be accommodated but will require discussion with facility staff. The SLAC POC should make the judgement call as to whether a change is beyond the approved envelop.

After the Shift

​Data analysis is expected after the data is collected and before the next data set is obtained. This requires use of calibrated in advance detectors and developing/testing data processing algorithms before run time. This is uniform requirement for all the experiments and excellent practice for science! Taking data blindly without understanding either its quality or its implications is unlikely to yield the best results. Subsequent beam time cannot be given without data analysis of previous beam time.

The procedure for the second shift will be expected after the completion of the first shift (should a second shift be necessary). As above, the procedure will be reviewed and passed to Jerry Yocky for scheduling.
 
(It is obviously advisable to have the second/third/fourth etc shift procedure prepared in advance and to revise it when lessons are learnt from beam time.)
 

8am the next day (or the next Monday for weekend beam time)

You are expected to send a representative to the 8am meeting. See MEETINGS section. 

It is possible for facility staff to be the representative but you must make this arrangement in advance.

Question:
How do I write a good beam time procedure?

Answer:
A good beam time procedure helps the operators know what to deliver and helps the experimenters to focus on achieving results. The more experience you gain, the more you will understand what constitutes a good procedure for you and your needs. But here are some general guidelines.
 
All procedures need to be headed with the expected date and time of the beam time.
 
The procedure needs to be sent to Mark and Carsten. The procedure will be reviewed and discussed with the experimenters if necessary (and it is usually necessary!). If approved, the procedure will be forwarded by Mark or a designee to Jerry Yocky who instructs the operators on how to prepare the beam.
 
1. A clear definition of what constitutes that the beam is ready for delivery to a given experiment.
 
This is a set of requirements on the
a) beam (charge, orbit, spot size, bunch length) and
b) required available diagnostics/hardware (TCAV…).
 
With regards beam parameters, explain what parameters are the critical ones. Give a tolerance where possible and indicate what is really essential to get any progress at all in the experiment and what is desirable to get the best data.
 
If you need it, ask for it, but if you don’t, don’t.
 
We do not want to get in a habit of saying we need something and then saying “well, I guess that’s good enough, we’ll take it”. At the same time we don’t want to hold off an experiment tuning on a parameter that isn’t really needed.
 
Although not needed within the document, be prepared to justify the requirements from a physics perspective.

 
2. A simple description of the measurements to be made, devices/parameters to be changed and rough durations.
 
Think of this as what you will go over with the operator before starting the measurements, e.g. “We will be measuring ___. To do this we will change ___. While we do this we need ___ to be monitored/held fixed. This will take approximately ____ hours.”
 
If the measurements require changing phases, magnets, waist locations etc it should be clearly listed.
 
 
3. A plan for the experimenters themselves.
 
This is an outline/plan to help the experimenters be ready to take data efficiently and is an extended version of (2). This document is for you the experimenters. Although this document will benefit from review and discussion with FACET management (Mark, Christine), in the end this is ‘for you’.

 
4. A definition of success.
 
This should define what subset of analysis will be completed quickly to understand what has been accomplished.
 
This analysis is important to present at the 8 o’clock meeting as well as to communicate what has been seen with management and operations.
 
Think of this as a plot(s) or image(s) that are not (necessarily) ready for publication but can illustrate the data. This last point is important for two reasons:
1. Guide what changes, if any, are needed from the accelerator and
2. Discuss how much more beam time, if any, is needed to move forward or complete the measurement.
 
Please be somewhat specific as to the date/time analysis will be ready as that would help us schedule a meeting if one is required.
 
Examples can be found in this shared google drive (feel free to use this to add your own procedure).

Question:
​What happens if I don't have an approved procedure for my shift?

Answer:
Beam time, even parasitic work, won't go ahead without a written procedure that has been approved by FACET management. This includes things like software tests with beam and controlling items remotely.

Question:
Why can't I do something that I have done before?​

Answer:
Just because you've done it before, doesn't mean it's okay to do it again. Each procedure is evaluated for that particular period of beam time. So if you want to do something that was in an earlier procedure and it was approved back then, unless it was approved in your current procedure, don't do it.

Conditions may have changed or lessons learnt. So please don't assume it is allowed unless explicitly okayed!

Question:
​What if I need to do something not on the procedure?

Answer:
We understand that plans do change.

Changes to beam time procedures should be directed to Mark (call him night or day) or his designee who can evaluate the impact and approve or not.
 
If a change happens during beam time, initiate the call.
 
If you ask operations to do something outside of the procedure, they should call us too but they may not so please don't hesitate to involve us to communicate with ops to confirm changes to procedure.

Question:
​Will I get beam time during the day or during the night?

Answer:
Summary:

FACET User shifts of beam typically begin between 2pm and 7pm and run for up to 12 hours. Multiple experiments may be scheduled for the same shift, running in series.

Detailed Daily Schedule:

Around 8/9 am each morning, beam physicists and accelerator operators set up the beam or engage in machine development studies to support delivery to users.

Beam is usually ready for delivery in the afternoon. 4pm is a typical start time. Some experiments that require the best stability won't begin until 7pm.

Preferably, all experiments scheduled for this block/shift are done by 4am and the accelerator operators perform standard maintenance and activities that improve the beam delivery for the next day.

At the latest, experiments should end by 7am which is when stability starts to be poor. Ending later also impacts the ability of the accelerator operators to set up for the next day.

The exact number of hours that an experiment gets is pre-arranged. Experiments are requested to keep to their time-slots so as to not affect other FACET users. Any changes, for example beam time extensions, need to be approved by FACET management.

See the image below.

Question:
When and what are the 8:15AM Meetings?

Answer:
A daily 15 minute meeting is held every day at 8:15AM on zoom (info https://confluence.slac.stanford.edu/display/FACET/Zoom+information​) while the accelerator are running. This meeting shows the machine performance in the past 24 hours, the results of an machine development and short reports from the experiments. This meeting is open to all and the audience stretches across many groups including beam operators, management and machine physicists. The Monday morning meeting also gives a report for the entire weekend and is usually heavily attended.

All FACET experiments that have received beam time are expected to send a representative to the 8:15AM meeting the next day (and also to the Monday meeting if they run between 8AM Friday and 8AM Monday). More than one representative is not necessary.

The representative is expected to report on the beam time. The report should only be a couple of minutes though important results would certainly justify more time! Some experimenters bring slides on their laptop though it is also good to just refer to posts in the FACET e-log.

A good report includes a comment on what the goal was and whether it was achieved, thanks to the staff for their support and if a next shift has been scheduled, what is coming next.

Question:
How do I get approved for beamtime?

Answer:
​See  SCHEDULING section above

Question:
What do I do on the day of my beamtime?

Answer:
Day of beam time
Jerry Yocky or designee passes the procedure document for the operators on to the people on shift so they can set up the beam and check out the hardware as requested in the procedure.
 
Approx 1-2 hours before scheduled start of shift
Check in with your facility POC or the ACR to ask what time delivery will begin. Sometimes the start of the shift can be delayed due to hardware issues that bring down the beam or make tuning difficult.
 
Start of Shift
Just before or at the start of the shift, you are requested to nominate a person in your group to act as the Point of Contact for communicating with Main Control/ the operators. This POC should talk through to procedure with the operator on shift for FACET so the operator has a good sense of how best to support the experiment.
 
The POC should check with the operator whether the requirements in the procedure were met. You are encouraged to ask for documentation which should all be posted in the FACET elog. (eg. a plot of the orbit, dispersion etc).
 
During the Shift
If the procedure changes during the course of the beam time, these deviations should be checked with FACET staff (FACET user manager, division director, science director etc.). This includes if tasks take longer than expected and the agreed upon length of the shift needs to be extended- this can usually be accommodated but will require discussion with facility staff.
 
This also includes tunnel access requests.
 
Maintain communications with your operator. Do not leave the ACR without telling the operator where you are going and how to contact you.
 
The facet elog must be used. Please see the facet elog FAQ above.
 
At the end of the shift
At the end of your beam time, the experimental team on shift must put in a summary. This really helps communication with the users, accelerator physicists and accelerator operators. We also use the reports from you to write the reports to DOE about how we are using beam time.
 
To generate a summary go to https://aosd.slac.stanford.edu/facet-shift-report/index/ and fill out the form.
When you are satisfied with you report, click on "SUBMIT" and the report will be saved in a database and also an entry will be made in the FACET e-log. Please go to the FACET e-log and make sure that the summary was added.

Question:
​What do I do on completion of my beamtime?

Answer:
After the Shift

​Data analysis is expected after the data is collected and before the next data set is obtained. This requires use of calibrated in advance detectors and developing/testing data processing algorithms before run time. This is uniform requirement for all the experiments and excellent practice for science! Taking data blindly without understanding either its quality or its implications is unlikely to yield the best results. Subsequent beam time cannot be given without data analysis of previous beam time.
The procedure for the second shift will be expected after the completion of the first shift (should a second shift be necessary). As above, the procedure will be reviewed and passed to Jerry Yocky for scheduling.
 
(It is obviously advisable to have the second/third/fourth etc shift procedure prepared in advance and to revise it when lessons are learnt from beam time.)
 
It is likely that a meeting with Mark is needed to look at the analysis and plans for the next shift. It certainly makes it quicker to "review" if we meet face to face. If a meeting is not arranged, please contact him (just drop by offices or call cell phone).
 
 
8am the next day (or the next Monday for weekend beam time)
You are expected to send a representative to the 8am meeting. See MEETINGS section.
 
It is possible for facility staff to be the representative but you must make this arrangement in advance and provide adequate detail to them to report on the beamtime.

Question:
​How do I get more beam time?

Answer:
​For various reasons, you may need more beam time.

We build "float" into the FACET schedule such that we can accommodate requests for more beam time typically towards the end of the run.

In fact, you may find that you have been allocated less time than originally requested because we need to see early goals accomplished before scheduling the "stretch" goals.

If you anticipate needing and being awarded more beam time, please keep yourself available for the end of the run (discuss with the FACET staff).

When you have your early analyses complete, present the analyses and shift request (please produce a coarse outline) to your facility Point of Contact.