Home  Company Background  Case Studies  Hardware  Software  Services COTS Article for control Engineering magazine, (July 2003) The following article was written for control Engineering magazine looking at the concept of COTS or Commercial of the shelf Systems. Whilst this might sound like a lazy way of developing systems, it has in fact some key advantages. The biggest of which is reliability which in our industry is critical. Anyway read on and see what you think. cw June 2003 Introduction ROV Network Ltd is a UK based control systems consultancy specialising in underwater robotics and tooling. This is a global market covering everything from simple tools for academic research such as seabed or water samplers through to advanced Oil industry and military robots designed to carry out construction work or hunt for mines. Other applications include cable and pipeline ploughs and trenchers that bury fibre optic cables or oil pipes to protect them from fishing nets and anchors. Autonomous battery powered robots that search hundreds of miles of seabed the list of applications is endless. Our clients in the last 12 months came from the UK, USA, Europe, China, Russia and the Far East. But the market is truly worldwide. Problems The environment is appalling, Dealing with the massive pressures found 5 kilometres underwater is only a part of it. Construction work in places like the North Sea and the Gulf of Mexico go on all year round. Weather conditions on the surface are often bad and even subsea, the visibility is often measured in a few inches. Surface operations in the splash zone can see a five ton machine thrown around like a toy. Currents can exceed 3 knots and operators (Pilots) may need to weave their way round steel and rope downlines from the vessels, scaffold poles and debris lying on the seabed. Opening an electronics pod in a snow blizzard is always fun. Operators work with voltages up to 5kv or higher, signals measured in mV, digital and analogue electronics, computers and software, Hydraulics, mechanics, fibre optics, acoustics, electromagnetics, X-ray equipment and of course the ever present sea. Massive structures are lowered onto the seabed by crane barges and positioned by free flying Remotely operated Vehicles (ROV's). With special tools, these machines will then pick up the ends of pipelines and align them to Valves on the structures, inserting gaskets, nuts and bolts before tightening them to preset torques. They operate Valves and other machinery using manipulators controlled from the surface with a delicacy that has to be seen. Ten minutes later they may have to ram a 20 ton pile into position so the structure can be pinned into the ground. Piling hammers create shockwaves that will quickly destroy poorly designed electronics and hardware. In days gone by, much of this work was carried out by divers but in today's world the risks involved are unacceptable and even with the most advanced diving techniques, commercial diving rarely goes beyond 400mtrs. Most modern ROV's are capable of 3000mtrs and new systems are often rated for 5000 mtrs and over. Why? because in our hunt for new oil reserves, that's where we have to go. In some respects, the environment is as alien and as tough as outer space yet the industry cannot support the kind of funding even the simplest of rockets will get. The solution of course is to keep it simple. Complex equipment will not be reliable in this environment. Keeping it simple though is often not that easy, the straightforward task of putting a nut on a bolt can only be achieved when you know exactly where the nut is, where the bolt is where the holes are and can align all the pieces perfectly with bad visibility and a current that wants to drag you away. Every job has to be planned carefully before any structure or tool enters the water. The industry is highly competitive, there are many companies out there creating innovative tools and products and the race is to get your product to market first whilst producing the most reliable equipment possible. Successes are largely forgotten but screw ups will be remembered in the industry for years so even the smallest of jobs must work perfectly first time. The oil industry is particularly conscious of price and delivery. Companies like ROV Network ltd have built up a niche market where complete control systems can be designed, built and delivered in just a few weeks. Most of these tools are unique so the development costs cannot be amortised over a large production run. Even where a product like a Construction class ROV is being built with a production run of over 30 units over a three or 4 year period and a unit cost of up to $3,000,000 per system, every unit will be adapted to the clients own specific needs, no two units are identical. The upshot is that development costs and lead time will make or break a project. Locking a dozen highly skilled programmers into a room for 6 months won't cut it. We need to develop the base algorithms and MMI within a few days and have the completed software ready in time for the hardware appearing a few weeks later. There are other issues. There is a history in the business of projects that quickly become unsupportable within a few short years. Hardware is the classic headache where chips, boards and busses become obsolete. With software and hardware, the engineers will follow the projects so a typical programmer, engineer and even project manager may change companies 3 or four times in a ten year period. Long term support for unique projects is difficult when the fundamental historical knowledge base from a company has dissipated. Yet ROV's and tooling systems can have a life well in excess of ten years. Clients need assurance that critical hardware and support will still be around long after the manufacturer, engineers and software programmers have disappeared. Rate of technical change in our industry is as high as any other. Systems built today must be able to support new tools and technologies that will come into use over the next ten years. Clients need to be able to expand systems to cater for job based technical demands that they cant even begin to imagine. Force feed back rate manipulators, Dynamic positioning, High bandwidth acoustic imaging, remote diagnostics and control through satellite links, compatibility with equipment as yet un-designed. No operator wants to be ruled out of contracts due to compatibility issues on a system barely a few years old.. Of course the working environment also includes the commercial, political and safety/QA concerns too. Equipment has to be not only fit for purpose, compliant with all current and most proposed regulations or guidelines but it also has to look good. Our clients have to sell their services to people who are very aware of current trends in technology. They expect output compatible with Microsoft Office products, email and web browsers. They are well aware of the difference between an Analogue video tape and a Digital DVD disk in terms of usability and ease of processing. When the job you do is hidden by several kilometres of murky water, the product your client gets is written reports and survey information including video recordings. Their opinion is often swayed by the quality of these deliverables. Technical support offshore is critical. Oil reserves seem to be located in the most godforsaken corners of the world, places with poor communications, minimal transport facilities, awkward border customs, language difficulties, zero sources for spares and government requirements to use local unskilled labour in the team. Systems used must be easy to diagnose, MMI's need to be able to display different languages. Systems must hold enough spares to cover most eventualities without breaking the bank through over stocking. Every operator has stories of $200,000 a day vessels on downtime due to the lack of some $10 potting compound or a $100 connector. Solutions So designers of equipment for underwater use have to be very careful in their selection of the appropriate technology. There is no right and wrong way of designing systems, all the systems working out there are good, The bad ones quickly disappear. But some are very definitely more suited to the environment than others. The solutions we aim for are not meant to be clever they are meant to be simple. Right from the start we try to use standard protocols and busses with the emphasis on compatibility and longevity. The obvious example here is PC's, No engineer will claim that PC technology is the best of designs but due to the massive user base, you can guarantee that the technology will be around long after many industrial bus systems have disappeared. The other advantage is that next years hardware will be faster and cheaper than this years. God bless Moores law. The PC104 implementation of PC's has many drawbacks, Dismantling a stack needs a lot of care and I've never liked ribbon cables in noisy environments. On the plus side, they are extremely rugged and the cost of the boards is very low so supplying complete built and tested stacks as spares is commercially viable. The physical size allows smaller diameter housings to be used which are more suited to the high pressures of deep water. Every electronics engineer out there even in third world countries knows and understands PC technology. There are hundreds of PC104 manufacturers many with international stockists, We use a lot of Diamond Systems products. The quality and range is superb. When problems arise we get to talk to the designers and solutions are provided quickly. As with all companies, products change over time creating compatibility issues but we can create new drivers in a couple of hours with simple installation instructions. Our software is normally supplied on Flash disks so a driver can be emailed out and dropped straight onto the networked system by the operators. Or on satellite linked systems, we can do it ourselves from here in the UK. With software, We need to be compatible with the widest range of systems. A product from a US company SoftPLC Ltd allows us to quickly create very compact control systems using ladder programming. The product has a range of communications options from RS232, RS485 and Ethernet and a communications link can be set up to reliably transfer any amount of data with a single instruction. Obviously this type of programming has limitations but SoftPLC allow us to easily tie in new instructions written in C to cater for time critical situations or unusual algorithms. ROV Network has a library of modules both for hardware interfaces and applications such as notch filters, saving and reading set-up files and interfaces to intelligent sensors. Most of these routines were written purely to optimise existing SoftPLC code, It is an extremely versatile language. This alone is not enough to justify using this product but it has two other superb qualities. Because it's simple, it can be read and understood by other programmers. Trying to get to grips with someone else's "C" Code is usually a nightmare but a well written Ladder program can be very quickly figured out and it helps that the programs themselves are modular and very small in size. The biggest advantage of all may be unexpected. PLC programs including Softplc can be monitored and reprogrammed in real time. The software provides a window into the code so that final development and bug finding can be carried out live whilst the equipment is being operated. The reduction in development time and thus cost from this simple technique is incredible. On our systems, the software is loaded into the newly built hardware in the morning and by lunchtime we are debugging the drivers and by supper we are usually crossing the T's and dotting the I's. Then the client comes along and says yes that's good but can we do this and make that do this so our programmer will sit down at his desk and implement live changes to a system whilst in water function testing is going on. I sat offshore on a vessel in the middle of the Atlantic last year watching a software engineer having major problems with a heave compensation system. He would watch the equipment go into violent oscillation. Sit and read his program for a while, Make a change to it, compile it, blow an EPROM, Walk across the deck and install the EPROM to the controller and then go back and switch everything on again, wait for the system to load and carry out another test. Me I sat next to the coffee machine with a laptop, a TV monitor and a comms set, Listening and watching the pilots fly the vehicle, I have a window into the software, I can change parameters and code in real-time and I can even take over control of the system flying a 5 ton ROV around the seabed using my laptops touch screen interface. Having the right tool for the job makes life a lot easier. The software is not hardware dependent so it can be ported to any system that can be made to run DOS, only the drivers need changing. We have used it on VME and STD legacy equipment with out any problems. Most new designs require compact solutions. A three inch PC104 cube with 32 AI, 20AO 64DIO, 6 RS232/485, Ethernet, USB and a couple of quadrature inputs fits into a very small housing and costs relatively little. Everything we do underwater needs to be isolated so there are additional boards and modules required for this but again where possible we use off the shelf equipment such as Burr Brown or Opto22. This makes sourcing of spares easy even in the more far flung reaches of the world. We do build proprietary equipment but only where the equivalent does not already exist or is not fit for purpose. Every solution will have drawbacks. Although SoftPLC boots in about 3 seconds, the Operating system can take a bit longer dependent on the hardware used. It uses a hardware based dongle which although perfectly reliable, is difficult to mount and looks out of place. But these are small niggles for a product that brings years of development and utter reliability For low volume and unique products in a highly competitive market, where future changes and upgrades will be a way of life, this solution is hard to beat. So that's the base control system, On the surface ship, a control panel of switches and joysticks connected directly to a SoftPLC stack and communication with a network of 1 or more subsea SoftPLC controllers up to 5 kilometres away through fibre optic links bringing video, telemetry, Ethernet and analogue or digital sonar data back to the operators. Fast efficient and highly robust. In over 5 years of using the product we have not had a single critical failure that wasn't caused by water ingress and our clients appreciate the simplicity and ease of use. The design is simple, spares easily obtained and a projected life of well over the ten years we are looking for. Should an interface card become obsolete, operators have dozens of choices for alternatives and the driver module can be rewritten by any competent programmer. What of the graphical MMI. We've already touched on the control panels but operator displays are critical. Not just for monitoring conditions and navigation but for diagnostics and fault handling/backup systems. The standard solution is once again a team of highly skilled programmers locked away developing screens from Visual basic or C++. One off projects though cannot justify this expense nor the potential for costly mistakes. Modern SCADA systems have developed into incredibly powerful packages with bolt on applications for alarms, logging, diagnostics, communications, language conversion, fault tolerant data channels and a host of extras such as Web browser interfaces and Dcom/OPC and dde communications. Whilst the unit cost for these packages is high, remember that 10's of thousands of hours have gone into their creation cutting the end developers time from months to a few days. With the use of a good OPC server such as Kepwares Kepserver programming becomes a mixture of desktop publishing and point and click visual programming. There are many good SCADA systems out there, The system of choice for us is Iconics Genesis. Again a highly respected product with a development team that are both knowledgeable and most importantly accessible. In the past their engineers have given us home phone numbers and even interrupted Christmas dinner for a quick technical chat. Over the years, these guys have bent over backwards to provide solutions for our unusual requests. We want to overlay real time graphics onto a video picture from a satellite hook up at three in the morning from a rig in the south China seas, No problem. And this, for a small company like ROV Network, is critical, My clients want solutions yesterday and knowing I can draw on the resources of companies like Diamond Systems, Softplc, Kepware and Iconics lets us sleep at night. Our suppliers are not unique, there are many good companies out there supplying off the shelf items but who go that extra distance, with engineers who give a damn. who want your business and want to help you succeed. Be aware though that there are many rubbish products out there too. Famous names who couldn't give a toss whether your product works or not unless your going to buy unit quantities of 1000 or more and even then only let you talk to an engineer via email or through a witless salesman. I can throw a Rock well cant I……………….better edit this bit out I guess Is it the perfect solution, No not completely, It's a very large package with capabilities that we barely touch on for our projects and the soft licence system can be a pain when someone decides to defrag the disk. But again it's utterly reliable and scaleable from the smallest project to the biggest. Our systems are built with a common procedure every time. First we get the specification of what the client requires. Then we try to work out what the client actually needs. We create the structure and do a first pass at the system with mock up screens which are submitted for approval. We then get the client in, sit him down with the programmer and modify the screens and systems to their preference. What colour, how big, what position. None of this affects the underlying design but the client buy's into the product through helping to develop an interface that suits his purposes. Once he's gone we then tie the dynamic functions into the SoftPLC stacks. This operation simply links a readout or switch to an input or output on the stack through a drop down list of I/O created originally in Excel. Gains and offsets can be handled by any of the software packages, we use the OPC server for simplicity and to ensure that other clients on the network get the same data. The software is then loaded to the newly built hardware, debugged and tested. Then it gets interesting. It's easy to design a product in a nice air-conditioned office. The real test is how it works on the back of a heaving vessel in a force 5 with high humidity, vibration, shock loads, tired operators who don't speak English and their client standing behind them screaming his head off. It's now we find out what can go wrong and why that clever piece of programming isn't worth a damn. So we send a programmer and an electronics engineer out with it for commissioning. Whilst the operator is working the system, the engineers are sitting in the corner monitoring and adjusting the system to tune it and make it work in the real world. With feedback back from the end users, Language translations supplied by the people who actually use it and diagnostics designed to cope with the reality of the situation. (What is the Chinese for "Nut Loader carousel deploy") All our systems are open source, We train the operators not only to use the system but how to modify it to add additional lockouts and visual interfaces. If operators don't like your system, it's the kiss of death and again by letting them change the interface to suit their needs without letting them into the core, it very quickly becomes their system. We take all that knowledge back with us so the next system is even better and when the client needs an upgrade, we're there to help. A recent trip to Angola for a system designed three years ago found major changes to the interface and a warm welcome from the operators. Their 20 ton tooling systems have over 1500 I/O and over 30 pages of touch screen controls alone. We upgraded the SCADA to the most recent version giving them increased capability and reliability. What was most impressive was the extent to which they had changed the system, tying in visual basic routines to the SCADA software to provide extended functions. The use of Commercial off the shelf software and hardware provides ROV Network with the lowest cost fastest development time solutions available. We can beat the pants off of most large developers not by days but by weeks and months, not by hundreds of $ but by 10's of thousands of dollars. Our Clients are comfortable with the product. It looks and is at the leading edge of development but has the reliability of established designs. They can modify it themselves or bring us or anyone else in to do it for them. They get longevity and easy sourcing, simplicity with extreme ruggedness and a future upgrade path. The use of Windows technology for the graphics is not to everyone's liking but remember that the graphics are not integral to the control, Should the top side graphics fail, the controller is unaffected. He still has cameras and sonar for feedback. Should the top side controller fail, the graphics system can talk directly to the subsea pods to allow an emergency recovery. (did I mention how versatile these solutions are?) Are these systems as reliable as a purpose built system. Yes, in our opinion they are more reliable. The systems we design are by nature intricate. The COTS products we use have taken decades to develop with 100's of man years of engineering. They are used in control solutions worldwide and have been for years. If they have a bug, you better believe that someone found it. That hand crafted system cannot be proved reliable until it's been properly punished for a few years. Does it have drawbacks. Well yes of course it does, It cuts into my profits for one, These products are not cheap solutions. They are also not proprietary solutions. Our clients can take the product and run with it themselves or go to any other company for support. Although our clients are keen on the idea, being open with your technology is a risky business just ask Apple computers….. Mmm maybe a bad example there. It may be possible to design systems that are smaller or quicker or cheaper or more reliable but you can only get one of those options at the expense of at least two of the others. The use of COTs technology for our industry gives the best all round reliability, performance and price. Chris Ward ROV Network Ltd 62 Murray Terrace Aberdeen AB11 7SB +44 1224 595555 +44 07979 307932 chris @ rov.co.uk As a footnote to this article, I wish to point out that I have no commercial links to any of the companies mentioned here beyond the fact that I give them money and they supply me with products and backup. I do however consider many of the people who work for them friends despite the fact that I have never had the opertunity to meet most of them face to face. So to Clive and Andre, Cindy and Dick and all the other great engineers out there that make me look good, and my company one of the very best, many many thanks.