Reason for change
24 February, 2003
1.1 THE PURPOSE AND
FORMAT OF THE STUDY
1.2 MAIN ELEMENTS OF THE
2 The Subaru Computer Architecture
2.2 SYSTEM SOFTWARE
3 The Subaru Software Subsystems
4 Future Observing Scenarios
4.2 NOD & SHUFFLE
4.3 SERVICE OBSERVING
4.4 ADAPTIVE OPTICS
6 TSC Improvement
6.1 ADDING A SMART
6.2 REPLACING THE
7 Concluding Remarks
7.2 SUMMARY OF
7.3 SOFTWARE ENGINEERING
Observatory Control Systems in General
8.1 Observation Control
8.2 Telescope Control
8.3 Instrument Control
8.4 Quick-Look facilities
8.5 Data Handling System
8.6 Logging and Fault
8.7 Data Reduction
In this report we examine the control
system software of the National Observatory of Japan’s 8.2 metre Subaru
telescope and compare it with the equivalent systems used by other similar
telescope projects such as Gemini, ESO VLT and Keck. We have paid particular
attention to areas in which changes might result in significant improvements in
observing efficiency and where the existing Subaru systems might struggle to
support future instruments and observing modes.
The study was carried out during
February 2003. The authors spent about a week on the Big Island,
received a series of presentations and observed the operation of the
telescope with SUPRIMECAM, FOCAS_MOS and ICRS (both with and without the adaptive
The systems directly involved in
observing, namely the telescope control system and observatory control system,
were examined in some detail, the data archive and data analysis system less so
and the instrument control systems—which are different for each instrument—not
The report begins by identifying the
main elements of the Subaru software, mainly for background but also with some
general comments. It then goes on to look at the architecture of the system—the
various levels of computer and networking—before dealing with the software in
detail. A section on future trends introduces a set of suggestions and
recommendations for change, and the report concludes with a summary of the
possible ways forward.
A checklist of the main features
needed in observatory control software is given in an appendix and referred to
at various points in the report.
The telescope control system (TSC) is
in direct control of the telescope; it makes the telescope and enclosure slew
and track, controls the instrument and image rotators, guides the telescope,
and maintains the figure and alignment of the telescope optics. It also handles
the many engineering data logging and maintenance functions required to operate
the telescope hardware.
The Instrument Control Software
controls the instrument hardware and detectors. Each instrument has its own
control system and, apart from using a common toolkit for the interface to the
observatory control system, does not have any software in common with other
The Observation Control System (SOSS)
coordinates the operation of the telescope and instrument in order to collect
astronomical data. It has an interface with both the TSC and the instrument
The Data Archive (STARS/MASTARS/SMOKA)
stores all astronomical data and provides facilities for searching for and
retrieving the data and implements the observatory’s data access policies.
There are three archives, one in Hilo (STARS), a copy in Mitaka (MASTARS), and
the archive of publicly available data (SMOKA).
The Data Analysis System (DASH) is a
system for implementing automatic data reduction pipelines and providing
offline data analysis facilities to users.
The principal deficiencies reported by
Subaru management and staff were:
Software maintenance, outsourced to
Mitsubishi and Fujitsu, is expensive and response to change requests is slow.
Observing efficiency is poor, at least
for some instruments; dithering performance is especially slow.
Users find observing preparation
arcane and the online user interfaces too complicated and inconsistent.
DASH is complex and little used.
The computer hardware chosen for
workstations (SUN sparc except for the TSC workstation which is a DEC alpha)
and mid-level processors (DEC alpha VME) are all widely deployed across the
computing industry and are unlikely to cause maintenance problems in the
foreseeable future. Replacement with more powerful or cheaper-to-maintain
systems should be relatively easy.
The local control units are based on
industry standard processors (Motorola 68030 and PowerPC) but are in any case
unlikely to need upgrading and replacements for failed units will be readily
The overall hardware architecture is
similar to other 8 metre telescopes, the only unusual features being the use of
RS422 for communications between the MLPs and LCUs and the location of the MLPs
and LCUs in a computer room rather than on the telescope structure. The number
of VxWorks systems is rather smaller than on other comparable telescopes (e.g.
more than ten on Gemini and hundreds on each VLT unit telescope).
The operating systems used on
workstations (SUN Solaris except for the TSC workstation which runs DEC OSF/1)
and mid-level processors (VxWorks) are all widely used in astronomy and
elsewhere and enable easy migration to new hardware (e.g. to Intel PCs
for workstations and PowerPC for MLPs) if this becomes desirable for
price/performance reasons. Migration to another Unix workstation operating
system (e.g. Linux) would probably also be possible with too much
The inter-processor communication
protocol (except between the MLPs and LCUs) is the SUN RPC (Remote Procedure
Call) which is in widespread use, is fully documented in publicly available
documents and is implemented on most operating systems.
The network infrastructure is all IEEE
802.3 (ethernet) and can be readily and cheaply upgraded to higher bandwidth
without impacting the rest of the system.
The Subaru Software Subsystems
The TSC performs its primary role of
tracking the telescope and enclosure, controlling instrument and image rotators
and auto-guiding competently. It is helped in this task by the excellent
performance of the telescope itself, which points well and has exceptionally
good open-loop tracking. However, some important features (see Section 8.2 in the Appendix) are absent, and certain aspects of
the TSC are slower than they should be; we concluded that software improvements
are needed in a number of areas, as follows.
The mid-level processor passes mount
and rotator tracking demands to the corresponding LCUs every 2 seconds. This
results in a delay of up to 2 seconds before the telescope responds changes in
target position. This is a serious limitation and is implicated in the present
unacceptably slow dithering performance (cf. 8.2.24).
The TSC has no provision for
automatically placing the target object at nominated coordinates in the focal
plane (cf. 8.2.15) other than the optical axis (normally coincident
with the image or instrument rotator axis). This means that the only mechanism
for “offsetting” the telescope—for example to centre the object on a
spectrograph slit—is to describe the offset in right ascension and declination.
This has a number of undesirable consequences:
The object position reported by the
telescope, and possibly recorded with the data, will be incorrect.
The necessary offset can be awkward to
calculate. For example, offsetting by a specified number of pixels on the
detector requires a calculation involving the (α,δ) of the target,
the rotator position angle and the orientation of the instrument with respect
to the rotator; furthermore the result is time dependent.
Subsequent adjustment of the field
orientation will cause the object to move on the instrument, necessitating
re-acquisition and hence costing time.
Trailing of images due to field
rotation occurs when tracking within 5 degrees of the zenith. The
telescope should be capable of maintaining accurate field orientation to within
0.5 degrees of the zenith, so this problem may be a software issue
probably related to the influence of the telescope pointing model on the field
orientation (cf. 8.2.13).
The TSC should warn the observer when
the object being observed will pass through the zenith blind spot. We noted
that while tracking through the zenith region not only was no warning given but
the reported limits were at times erroneous (cf. 8.2.13).
There is no provision for taking
account of a difference in effective wavelength between the guider and science
instrument (cf. 8.2.6). This causes trailing of images during long
exposures due to changes in refraction. This “atmospheric dispersion” or
“chromatic differential refraction” effect can amount to several arcseconds.
The auto-guider supports just one
guide star acquisition mode (cf. 8.2.14): when guiding is turned on the guide star is
“frozen” at its current position. This means that blind acquisition of
invisible objects relies on the (exceptionally good) open loop tracking of the
telescope. This may not be adequate for the very small slit sizes made possible
by adaptive optics and in any case requires that a nearby visible object is
acquired first, a potentially time-consuming operation. An acquisition mode where
the guide star is pulled to a position on the guider CCD calculated from the
guide star catalogue position would enable acquisition in a single operation,
with an accuracy limited only by the quality of the guide-star and
science-target coordinates and the precision of the guide probe positioning
(the latter is not very exacting given the large plate scale on an 8 metre
Pointing-test observations are logged
in a “relative” form that means that analysis requires knowledge of the
pointing model installed at the time the test was run. This can cause
difficulties when analysing historical data to detect long-term trends; it
would be better to log absolute position and time information (cf. 8.2.17).
We saw maps of pointing calibration
stars on a regular (α,δ) grid. This biases the pointing model to the
region around the pole, which is not desirable. Test points that are equally
dense all over the sky would be better (cf. 8.2.17).
Adding new terms to the telescope
pointing model requires modifying, or adding to, code in the TSC. This is a
problem for Subaru because of the long turn-around for software modifications
by Mitsubishi. Also, it has led to an approach where azimuth track effects are treated
with a separate model; a single pointing model that includes both these effects
and the conventional geometrical terms could deliver better results.
The SOSS skeleton files implement an
exceptionally powerful and flexible language for sequencing concurrent
operations on the telescope and instruments (cf. 8.1.3). The code is not particularly elegant, but probably
does the job of controlling the telescope and instrumentation as well as
anything that could replace it.
However, there is a lack of
“astronomer friendly” tools for generating observing scripts (cf. 8.1.1), and this has given SOSS a bad name with users.
Preparing for an observing run is a laborious and error prone process for the astronomer
(and the support scientist as well). As well as preparation tools, script
validation tools are also needed to avoid errors that waste telescope time.
The user interface presented to the
support scientist running the observations is at heart a command line system,
supplemented by a rather crude GUI. The observers find the GUI unwieldy and
prefer to use the command line interface even though the only editing facility
for the rather long and complex command lines is to cut-and-paste from
previously executed commands. What would be better is a set of graphical
interfaces organized with the observer in mind, with commands grouped by
function and reflecting frequency of use.
The skeleton files (for ICRS at least)
contain numerous “sleeps” (interval waits), typically of 1 second. This is a
bad sign and may be one reason for the very slow performance of the system.
Better understanding of the real-time properties of the system is needed.
We did not examine DASH in detail so
these remarks are based solely on material presented to us. We noted that the
system is comprehensive and uses modern software techniques. It is
forward-looking and should blend well with future “Virtual Observatory”
DASH has not been much used so far in
its offline data analysis role, because astronomers find it difficult and
complicated in comparison with well-established and familiar interactive data
reduction tools such as IRAF.
However, its intended role in
providing an automated data reduction pipeline is another matter. Although
little used at present, this pipeline function will increase in importance as
the observatory moves away from “classical” observing, where the astronomer is
present at the telescope, and moves towards queue mode and service observing
where the telescope is operated by Subaru staff on behalf of the astronomer.
The ability to do at least preliminary data reduction in some automated way
will become essential in order to ensure that the data provided to the
astronomer are of the highest quality.
In short, we believe that an automated
data reduction pipeline is an essential ingredient of a successful service
observing operation and there is no reason why DASH should not fulfil this
requirement. Writing the necessary data reduction procedures will require
significant effort but will result in savings in the long term.
We did not examine the data archive
systems in detail so these remarks are based solely on material presented to
The Subaru data archives compare well
with the archive systems being used by other optical/infra-red observatories.
The support staff have a clear vision of where improvements are needed and how
these can be achieved.
Subaru is a good example of what can
be achieved by the Japanese government’s policy of forming collaborations
between industry and academia: Mitsubishi and Fujitsu have done an excellent
job of implementing what the Subaru designers asked for. However, the
consequent outsourcing of software maintenance to Mitsubishi and Fujitsu has
become a serious limitation on what the Observatory can do and in what
timescales. It is essential to bring about circumstances such that
(i) Subaru has full access to source code for all the
non-industry-standard software it uses, and (ii) the Observatory can make
its own revisions and bring them into use without delay.
In this section we discuss a number of
observing modes that are likely to become important in the future and that
illustrate how some of the deficiencies outlined in Section 3 could adversely affect future development of the
Nod and Shuffle
is a technique recently developed for improving sky subtraction on optical
detectors limited by readout noise. It involves simultaneously moving the
telescope and the charge on the detector. Implementation of this technique
requires that the telescope offsets exactly match the pixels on the detector.
This is extremely difficult to achieve if the telescope does not support
offsetting in the focal plane, as is the case at present with Subaru.
It is generally accepted that making
the most efficient use of today’s large telescopes means moving away from the
traditional mode of operation where individual astronomers are allocated a
fixed period of telescope time towards “service observing” where astronomers
apply for data rather than telescope time. The observations themselves are
carried out by staff astronomers, who make decisions about which programs to
execute, and when, in order to maximize efficient use of the telescope and make
best use of the prevailing conditions.
In order to run such an operation
efficiently it is essential that observing proposals contain all the detail
necessary to allow the observations to be carried out without reference to the
requesting astronomer. This can only be achieved if the astronomer has access
to easy-to-use tools for generating the observing proposal.
Efficient operation is only possible
if observing programs are free from errors at the point that they are executed
on the telescope. This requires that the support scientists have tools
available to them for validating programs against the currently available
instrument configurations (available filters, gratings etc.) before placing them in the queue for
The telescope control system must take
account of the astrometric changes introduced by AO systems, such as the
effects of atmospheric dispersion compensators and, in the multi-conjugate case,
the need to predict changes in the relative positions of three natural guide
stars to better than the diffraction limit.
The protocol for sending tracking data
from the MLP to the mount and rotator LCUs should be redesigned to eliminate
the 2-second latency. The present rigid “timing diagram” view should be
replaced with a flexible approach where determining the next demand is
decoupled from “just in time” samples of the tracking locus. The aim should be
to make the full control bandwidth of the telescope mechanics accessible to
The TSC should be upgraded to support
an integrated approach to rotator and guider control and focal plane management
such as that provided in the Gemini TCS and in the TCSpk pointing kernel.
How to go about this is complicated by contractual issues with Mitsubishi and
the possible non-availability of the source code of the current system;
possible ways forward are discussed in Section 6.
High priority should be given to the
creation of an observation preparation tool, because neither writing a tool
from scratch nor adapting an existing tool from another observatory to support
Subaru’s instruments will be a rapid process. The output from such a tool would
be the observing scripts that are currently generated by hand; this means that
the interface to SOSS is already defined and consequently it would be feasible
to contract out the development work. Section 8.1.1 in the Appendix lists some of the functions required.
The cost of developing a preparation
tool is hard to estimate because it depends so much on the level of
functionality required. A simple but nonetheless useful “form-filling” GUI could be developed for
perhaps as little as 1 staff year plus 0.25 SY per instrument. On the other
hand a fully featured tool such as the Gemini Observing Tool could cost 20 SY.
Before embarking on development, we
recommend that Subaru examines a representative sample of the preparation tools
in use by other observatories and explores the possibility of adopting an
existing tool. Gemini, the Joint Astronomy Center (JAC) and ESO are all
candidates. If feasible this approach could yield considerable cost savings.
The problem of the slow operation of
SOSS needs to be addressed by a programme of measurement and analysis in order
to understand where improvements are both needed and possible. There is
unlikely to be one single cause so early dramatic improvements are unlikely. In
particular we would not expect CPU speed or network bandwidth to be a primary
limitation, and advise against any plan to upgrade hardware before the real
causes of the slow performance have been identified.
The SOSS software is maintained by
Fujitsu, leading to high cost and slow response to change requests. However,
most of it functions well, and a complete rewrite would probably not be
cost-effective. What is needed is acquisition of the source code and the
development of a Subaru in-house capability to maintain and enhance the system.
Developing new user interfaces to meet the needs of Subaru support astronomers
would be one such enhancement; it is essential to get into the position where
good ideas can quickly be put into practice and new observing modes tried out
as the need arises.
The current archives appear to be well
able to support the requirements of the observatory and we see no need for any
changes in this area (cf. 8.5.1).
Future development of DASH should be
focused on the needs of service observing, including the development of the
pipeline data reduction capabilities (cf. 8.7). It would also be useful to look into the JAC
ORAC-DR system as an example of what can be achieved in this area.
The biggest single improvement to the
current system would be to increase the degree of automation of the TSC,
offering a smarter interface to instruments. We have confidence in the
astrometric accuracy of the present TSC, but it is clear that moving certain
associated computations, in particular the (x,y) to (α,δ)
transformations, out of the higher level software and instruments and into the
TSC will pay dividends, delivering pointing integrity and making slick and
accurate instrument applications possible.
A TSC upgrade of this sort could be
done so as to preserve the existing command interface, offering all the current
(low-level) capabilities. Consequently the upgrade would be transparent to
SOSS, which could use the new TSC without change. New commands would be added
to the TSC interface in order to give access to the new functionality, and SOSS
could be modified to take advantage of these as required.
The cost of upgrading the TSC with a
new, integrated, pointing kernel is hard to estimate since it depends a great
deal on the internal architecture of the current system; it might, for example,
be equivalent to 2-3 staff years to provide TCSpk and the associated RAL kernel
and develop interfaces to the Subaru systems. The work would require
cooperation from Mitsubishi, including disclosure of TSC source code.
Replacing the TSC entirely would be
more expensive and take longer but could, if necessary, be done without access
to the source code of the existing system. It would of course mean abandoning a
large amount of perfectly satisfactory code and replacing it with new code with
no improvement in performance except in specific areas.
We would expect such a plan to involve
replacing the code that currently runs in the TSC workstation and the MLPs but
not the code in the LCUs (except perhaps in order to implement new guide star
acquisition modes). This means that the interface between the MLPs and the LCUs
would have to be disclosed—and documented—by Mitsubishi.
The cost of writing a new TSC is
probably of the order of 12 staff years assuming that existing astrometric
software such as the TCSpk kernel was used.
Subaru’s computing infrastructure is
in good shape. It uses mainstream components, standard in industry and
astronomy, which can readily be maintained and which offer relatively
inexpensive future upgrade paths.
The TSC competently realizes the
superb optical and mechanical capabilities of the Subaru telescope, but lacks
important high-level interfaces and has some serious speed problems.
The SOSS skeleton file system is
powerful and flexible. Although the scripting language itself is rather
old-fashioned in style, any replacement would look just as arcane.
The interactions between SOSS and TSC
seem to involve significant latencies that could explain some of the slow
performance. This and the way the TSC tracking demands are managed are a more
likely explanation of the slowness than hardware limitations such as the speed
of CPUs and LANs.
Friendlier interfaces between
end-users and SOSS are required.
DASH is a good system with much in
common with other such systems being developed elsewhere. However, its immediate
importance lies in supporting pipelines at the telescope more than in offline
The data archive facilities appear
competent and the plans for future development sound.
Future observing modes will stretch
the system unless it is enhanced in certain ways.
The present outsourcing of software
maintenance is holding Subaru back (as well as being expensive).
After first looking at what other
observatories have done, Subaru should develop an astronomer-friendly observation
preparation tool to front-end SOSS. This should support classical observing,
with the telescope user participating, but should be aiming towards a future
where service observing is the norm.
The present extremely slow interaction
between SOSS and the telescope must be properly diagnosed and then put right.
It will not be enough to upgrade the computers and networks involved.
A smart interface to the telescope
should be developed, providing full pointing integrity and fully automatic
acquisition of faint objects onto instrument slits.
The most difficult task facing Subaru
is the transition from entirely outsourced software maintenance to an in-house
capability. At an early stage these long-term possibilities must be discussed
with Mitsubishi and Fujitsu with the intention of securing their cooperation
and optimizing their future roles. Some detailed points concerning management
of in-house software activities are given in the next section.
Forming a Subaru in-house software
team will require a wide spectrum of people from astronomers to software
specialists. The team members must work together (i.e. not just
hierarchically), with flexibility and discipline carefully balanced. Knowledge
must be shared freely, and single points of failure guarded against.
Individual engineers should develop
expertise in particular techniques (e.g. real-time control,
user interfaces, data formats and world coordinate systems, documentation and
version control) rather than be allocated to specific items of hardware (e.g. telescope,
autoguiders, individual instruments), and there ideally should be
considerable overlap so that no-one is indispensable.
To tackle major new developments, we
counsel Subaru against the orthodox “big design up-front” by astronomers
followed by a long implementation phase by software engineers; this is rarely
successful in practice simply because the first design is never the right one.
But the other extreme, namely ad hoc and uncontrolled development in
response to the most recent operational pressures, is equally dangerous. In
practice, the most successful projects involve all the team members in a
thorough initial design, focusing on interfaces between subsystems, followed by
rapid prototyping and then a long iteration phase with both the end user and
the software engineer in the loop at all times. It is helpful throughout the
development phase to have a programme of actual deliverables so that progress
can be assessed. The existing Subaru preference for using industry-standard
tools in commonsense ways should be preserved, given the long service life of
the systems concerned.
Strict control over versions is vital,
and documentation must be kept up to date. To maintain standards, all code and
documentation must be checked for quality by someone other than the author.
The authors would like to thank
Hiroshi Karoji for the opportunity to study the Subaru systems and for his
hospitality and encouragement. We greatly appreciated Mark Weber’s meticulous
preparations for our visit, which meant that the limited time available was
used very efficiently. Finally we would like to thank all the speakers for
their detailed presentations and for
their openness and insight, and the observers, instrument scientist and
telescope operators for their patience in answering our many questions.
In this appendix we itemise the functions required of a modern
observatory’s software systems (but excluding finance and “office-automation”
functions, and data analysis software used by observatory staff for personal
Access to catalogues of stars and
other astronomical objects
An exposure calculator for estimating
A way of selecting suitable guide
A way of selecting instrument
configurations (filters, gratings etc.)
An interface for defining patterns for
An interface for defining the instrument
Observing Database that
Tracks status of observations
Support usage accounting
Records configuration of actual
Keeps observer informed of progress
Supports scheduling decisions
Generates observation descriptions to
be executed on telescope
Steps through the operations needed to
carry out the observation.
Allows appropriate operator
intervention during execution.
Enables last-minute modification of
Updates observing database with status
Updates observing database with actual
Accepts celestial targets in a variety
FK5 mean place
FK4 mean place
Apparent place (geocentric and
topocentric, equinox and CEO)
Proper motion, parallax and radial
Exploits full performance of mount,
rotator and enclosure
Does not disturb optics
Tracks smoothly and with long-term
accuracy limited only by the pointing model
Rotator stabilizes field orientation
to place a specified instrument direction at a specified position angle with
respect to a specified celestial coordinate system
Enclosure follows telescope
Total pointing integrity—all
components are part of one pointing context
ADC and AO integrated
Fully corrected pointing transparently
Tracks solar-system objects
Oversees maintenance of optical
Open loop models for M2 position and
Closed loop control with wave-front
Leap seconds properly handled
Current UT1-UTC & polar motion
taken account of
Accurate maintenance of field
Warns operator if current objects will
track through “at risk” region
Mechanism demands do not become
undefined in inaccessible region
Azimuth and rotator demands have
“sensible” trajectory when transiting near zenith
Automatic acquisition of catalogue
Automatic selection of guide stars
Guiding when guide star coordinates
not accurately known
Solar-system objects as guide stars
Guiding while performing differential
Windshake rejection through M2 and
Blind offset guiding using astrometric
Pointing refers to nominated
“hotspot”, not just the rotator axis
Quick selection of hotspot
Hotspot calibration procedures
When instrument is rotated, field
rotates about hotspot
World coordinate system delivered to
instruments linking pixels to RA,Dec
Ability to adjust pointing, target and
hotspot as appropriate
Feedback on what the telescope is
Display of times to wrap and other
Wind speed and direction
Humidity and dew-point
Automated pointing tests
Sky sampled evenly
Logging of pointing-test data
log everything needed for re-analysis
data independent of model in use
"Mini-test" procedure for
Pointing analysis and modelling
Alternate pointing models for
different telescope configurations
Management of cable wrap
Selection of wrap state when slewing
Calculation of times to limits.
Rotators at different foci
Chopping and nodding
Multiple ways to specify chop
Instrument synchronised with M2
Bandwidth splitting between mount and
“Dawdle” mode, to dither without losing
Management of offsets from base
Give access to full bandwidth of
Can be synchronised with instrument
Data can be recovered after error conditions
Displays data as soon as available
Enables preliminary measurements
Associates data with ancillary
Facility adaptive optics
Associates science data with
appropriate calibration data
Respects observatory data access
Collect performance data.
Measure “in service” performance.
Detect trends to pre-empt faults.
Analyse operational efficiency
Log fault conditions
Assist night time fault diagnosis by
operators and support engineers
Assist fault diagnosis by day crew
Analyse statistics to identify weak
Shares data analysis applications with
those available offline
Assists immediate data quality
Produces standard data products
suitable for archive
Efficient generation of
Requires minimal operator intervention
Robust error detection and handling