I work on equipment automation, the worst errors we get are intermittent.

One of my guys is working on a machine in Phoenix that manufactures computer wafers, we redid the robotics on it. The customer provided the initial program that we were supposed to modify, it had a lot of bugs in it that we had to deal with. The total overages were 280K on a 110K project that I had eaten, we reached a settlement with the customer and split the losses, some of the stuff was due to my guys not understanding, and some do to the bad program. I am OK with the settlement, and the customer has placed an order for more machines, with very defined scope that I should make up the rest on, and I will manage so we stay in scope.

This machine has a large (30 inch) carbon disk with five small (6") holes in it for silicon wafers. This is a lot like the old multi CD changers that had big disk with holes. The machine has to home the disk so the other robot can load and unload wafers. It would error out randomly about every 100 wafers. We had to devise a trap to look for these events and it caught the event this morning. We have to build traps sometimes when looking at a program just does not disclose the error.

We have been looking for this particular issue going on two weeks now.

It is miller time.

This one job is why I got clobbered financially last year.

David;
I feel your pain. I had an intermittent that took me over 3 months to find. It was caused by a micro switch not adjusted quite correctly. Fortunately, it was a repair job, so no loss for me.
Can you say what the problem was?

There is a recipe that is downloaded that controls the machine, somewhere in the logic the command from that recipe is not getting sent to the servo. I had suspected a loose wire on a sensor, but this finding rules that out.

I feel for you. Trouble shooting my business is about impossible. I try but 20 years into it and I still throw my hands up many days.

As soon as you said the customer provided the program and it had bugs in it......my spine went Tingley. To me that called for a debugging clause. Just my opinion. You can have one or hundreds of little bugs. Some of the stuff I used to work on even had stepper switches. When one of those misses a contact in the countdown program whoa nellie! Modern electronics and computer guided operations just go a heck of a lot faster and can be intermittent to the sky if it wants to.

I know how you feel. When Onan first came out with the computer controlled generator system. It was a real nightmare. Parts that were supposed to talk to each other didn't. Then they did or one panel refused to talk to the others. A startup that should have been a few hours. Turned into a day or two. Had you talking to yourself.

Which is worse - a costly error that is in your control, or a costly error that is outside of your control?

Under you control you eat the loss.
Outside your control, it would depend (always!) on how the contract was written.

In most construction work the bidder goes in as low as he can and still make a reasonable profit, and then hopes for change orders in order to really make money.

Had similar problems in the Air Force, they updated the Avionics system on the B-52H so it could carry cruise missiles. At the time we could do an alert upload in less than a half of the day that's from pre-load on the replacement bird to uncocking and download of the bird being taken down. The first bird we put on alert with the new system took us a week to get cocked on, every time we got the launcher in the aircraft and tested it it would fail, it tested good at the missile shop, and we could unload it from the aircraft and re-test it at the shop and it would pass, a lot of it was stacked tolerances so through trial and error we grouped missiles and launchers to counter some of the bias and then over the course of time kinda figured out which bombers worked with what launchers.

It goes around in all businesses it sure sucks to hear stories like that with a problem that isn't yours cost yo you a pile of money.
We try to step around the holes like that, but same as you took a solid whooping last year. We had 2 jobs that we're loser's 1 at a school and the other a time a potato chips plant. It's darn good thing the rest of the smaller jobs went well, that and service work kept us going. The school job was a comedy of errors by the general contractor and finding a pile of hidden asbestos removal that had not been done but was supposed to have been done years ago. it was nicely up behind hard cielings that were just covered over and not properly addressed during the original abatement. Fun times( not).

(quoted from post at 16:04:00 02/02/18) Had similar problems in the Air Force, they updated the Avionics system on the B-52H so it could carry cruise missiles. At the time we could do an alert upload in less than a half of the day that's from pre-load on the replacement bird to uncocking and download of the bird being taken down. The first bird we put on alert with the new system took us a week to get cocked on, every time we got the launcher in the aircraft and tested it it would fail, it tested good at the missile shop, and we could unload it from the aircraft and re-test it at the shop and it would pass, a lot of it was stacked tolerances so through trial and error we grouped missiles and launchers to counter some of the bias and then over the course of time kinda figured out which bombers worked with what launchers.

Click to expand...

Tolerance buildup an really kick your butt.Everything is within spec but it still won't work!