Intellige telligent nt We Weld lding ing and d We Weld lder - - PowerPoint PPT Presentation

Mo Modelling elling of

• f Hum

uman an We Welder er for

• r

Intellige telligent nt We Weld lding ing and d We Weld lder er Traini aining* ng*

YuMing Zhang University of Kentucky Lee Kvidahl Ingalls Shipbuilding NSRP Welding Panel Meeting Bethesda, Maryland May 4-5, 2016

For Unlimited Distribution. Research funded by the NSF under grant “NRI-Small: Virtualized Welding: A New Paradigm for Intelligent Welding Robots in Unstructured Environment,” IIS-1208420, Sept. 2012-August 31, 2016 and grant “Machine-Human Cooperative Control of Welding Process” CMMI-0927707, October 2009-Spet. 2013

Contents

• 1. Sensing of 3D Arc Weld Pool Surface: motivation, method, real time
• 2. Characterization: from numerous points to three characteristic parameters
• 3. Control of 3D Weld Pool Surface: control theory method
• 4. Human Welder Response: modeling and analysis, control using welder model
• 5. Welder Motion Response: human-robot system, speed adjustment, 3D adjustment
• 6. Future Directions

1. . Sen ensing sing of

• f 3D Arc We

Weld d Poo

• ol Sur

urface ace (Hu Human man Res esponse ponse Inpu put) t)

Weld pool: where complex phenomena originate; but only the surface is visible; the major feedback information available to human welders Measurement of weld pool surface temperature distribution: needs the emissivity to determine the temperature the infrared radiation but the emissivity is slope dependent Weld Penetration:

(1) Surface Specular: use laser reflection; (2) Arc Radiation: use laser reflection and intercept at a distance  low power continuous laser for continuous measurement, no need for a special camera Laser: 20 mW, 685 nm Y.M. Zhang, H.S. Song, and G. Saeed. Observation of a dynamic specular weld pool surface. Measurement Science & Technology, 17(6), 2006.

 Reflection law, surface constraint, error evaluation

• 40
• 30
• 20
• 10

10 20 30 40 20 40 60 80 100 120 140 160 180 X/mm Y/mm Reflected dots from image processing and reconstructed surface Image processed dots Dots reflected by reconstructed surface

Hongsheng Song. Machine Vision Recognition of Three-Dimensional Specular Surface for Gas Tungsten Arc Weld Pool. ECE Department, University of Kentucky, 2007. XiaoJi Ma. Measurement of Dynamic Weld Pool Surface in Gas Metal Arc Welding Process. Department of Electrical and Computer Engineering, University of Kentucky, Feb. 2012.

Analy alytical tical Solu luti tion

• n

Key for Real Time Measurement and Control

W.J. Zhang, X.W. Wang, Y.M. Zhang, 2013. “Analytical Real-time Measurement of Three-dimensional Weld Pool Surface,” Measurement Science and Technology, 24(11), article Number 115011 (18pp), doi:10.1088/0957-0233/24/11/115011

2. . Ch Char arac acterization terization of

• f 3D We

Weld ld Poo

• ol

l Su Surface ace

 Characteristic parameters should be used rather than a large set of 3D coordinates. Should keep the fundamental information in the weld pool surface about the weld joint penetration.

W.J. Zhang, Y.K. Liu, X. W. Wang, Y.M. Zhang. Characterization of three-dimensional weld pool surface in gas tungsten arc welding. Welding Journal, vol. 91, 2012.

Left: Measured 3D weld pool surface parameters from 36 experiments; Right: Least squares model fitting with 3-parameter model using the width, length, and convexity.

1.7906 0.5657 10.8057 0.9868

b

w W L C    

3. . Con

• ntr

trol

• l of
• f 3D We

Weld d Poo

• ol Sur

urface ace

Modeling: how the characteristic parameters respond to the change in current and travel speed – extract the model from experimental data Control: Model predictive control algorithm

Yukang Liu, YuMing Zhang. Control of 3D Weld Pool Surface. Control Engineering Practice, 21(11), 2013.

Welding Experiments

Speed Disturbance

20 40 60 80 100 120 2 4 6 Distance (mm) Wb (mm)

10 20 30 40 50 60 70 80 90 100 110 120 1 2 3 4 5 6 7 8 Time (s) Input Parameters Current/10 (A) Voltage/3 (V) Speed (mm/s) 10 20 30 40 50 60 70 80 90 100 110 120 1 2 3 4 5 6 7 Time (s) Weld Pool Parameters Width (mm) Length (mm) 10*Convexity (mm)

4. . Mo Modeling eling and d Analy nalysis sis of

• f Hum

uman an We Welder er Res esponse ponse to to 3D We Weld Poo

• ol Sur

urface face

(mechanized welding, human adjusts the current)

Welding Parameters Current/A Welding speed/mm/s Arc length/mm Argon flow rate/L/min 57~81 1~2 3.5-4.5 11.8 Monitoring Parameters Project angle/° Laser to weld pool distance/mm Imaging plane to weld pool distance/mm 35.5 24.7 101 Camera Parameters Shutter speed /ms Frame rate/ fps Camera to imaging plane distance/mm 4 30 57.8 Manual control system of GTAW process

 Skilled human welder holds the current regulator while observing the geometry of weld pool;  Adjusts the welding current to control the process to full penetration.

Experiment Parameters

Y.K. Liu, Y.M. Zhang, L. Kvidahl. Skilled Human Welder Intelligence Modeling and Control. Welding Journal, 93, 2014.

Linear modeling result.

 In general, the human intelligent model can be written as:

200 400 600 800 1000 1200 1400 1600

• 4
• 2

2 4 Sample Number dCurrent Measured dCurrent Linear Estimated dCurrent

( )= ( ( 3), ( 3), ( 3), ( 1))

f f f

I k f W k L k C k I k      

 Following linear model can be identified using standard least squares method:

( )= 0.16 ( 3) 0.082 ( 3)+1.81 ( 3)+0.26 ( 1)

f f f

I k W k L k C k I k        

Model comparison between linear and ANFIS model. Model Comparison between Neuro-Fuzzy Model and linear model

50 100 150

• 2

2 4 Sample Number dCurrent Measured dCurrent Linear ANFIS Estimated dCurrent 500 520 540 560 580 600

• 2
• 1

1 2 3 4 5 Sample Number dCurrent Measured dCurrent Linear ANFIS Estimated dCurrent 1360 1380 1400 1420 1440 1460

• 2
• 1

1 2 3 Sample Number dCurrent Measured dCurrent Linear ANFIS Estimated dCurrent

Average Model Error /A RMSE /A Maximum Model Error /A Linear Model 0.52 0.79 3.15 ANFIS Model 0.50 0.76 3.03

Y.K. Liu, W.J. Zhang, Y.M. Zhang. Dynamic neuro-fuzzy-based human intelligence modeling and control in GTAW. IEEE Transactions on Automation Science and Engineering, 12, 2015.

Model Comparison

Linear Models

( )= 0.049 ( 3) 0.0049 ( 3)+1.73 ( 3)+0.72 ( 1) I k W k L k C k I k         ( )= 0.16 ( 3) 0.082 ( 3)+1.81 ( 3)+0.26 ( 1)

f f f

I k W k L k C k I k        

Novice welder Skilled welder

Nonlinear Model Comparison

 In normal cases the skilled welder's adjustments are minimal which can prevent large oscillation and overshoot novice welder model suffers;  In other cases where the convexity is either considerably small

• r

large, the adjustment made by the skilled welder is larger than that of the novice welder, which can provide shorter settling time than novice welder does.  The skilled welder model does provide better adjustment than the novice welder.

Nonlinear model surface of the neuro-fuzzy human welder model (left: novice welder, right: skilled welder) for convexity = (a) 0.10 mm (b) 0.18mm (c) 0.26mm. Previous response is zero for all cases.

3 4 5 6 3 4 5 6 7

• 2

2 W (mm) Novice Welder L (mm)  I (A)

• 4
• 3.5
• 3
• 2.5
• 2
• 1.5
• 1
• 0.5

3 4 5 6 3 4 5 6 7

• 2

2 W (mm) Skilled Welder L (mm)  I (A)

• 3
• 2
• 1

3 4 5 6 3 4 5 6 7

• 2

2 W (mm) Novice Welder L (mm)  I (A)

• 0.2

3 4 5 6 3 4 5 6 7

• 2

2 W (mm) Skilled Welder L (mm)  I (A)

• 0.1
• 0.05

3 4 5 6 3 4 5 6 7

• 2

2 W (mm) Novice Welder L (mm)  I (A)

3 4 5 6 3 4 5 6 7

• 2

2 W (mm) Skilled Welder L (mm)  I (A)

• 0.5

(a) (b) (c)

Control Experiments: Varying Initial Current

Control experiment result with different initial current (a) 52A; (b) 54A.

A B

A B

10 20 30 40 50 60 70 80 90 100 110 1 2 3 4 5 6 7 8 Time (s) Input Parameters Current/10 (A) Voltage/3 (V) Speed (mm/s)

B A

(a) (b)

A B B A

10 20 30 40 50 60 70 80 1 2 3 4 5 6 7 8 Time (s) Input Parameters Current/10 (A) Voltage/3 (V) Speed (mm/s)

B A

• 5. We

. Welder der Mo Moti tion

• n Res

esponse ponse to to We Weld Poo

• ol Sur

urfac face

(weldi lding ng speed ed adjus ustment ment, , 3D opera ratio tion n adjustment) ustment)

1. Equipment for experimental data: human-robot system 2. Extract good response from not-perfect performance of human welder 3. Quality evaluation model 4. Supervised learning using good data

Y.K. Liu, Y.M. Zhang, L. Kvidahl, 2014. “Skilled Human Welder Intelligence Modeling and Control: Part I-Modeling,” Welding Journal, 93: 46s-52s. Y.K. Liu, Y.M. Zhang, L. Kvidahl, 2014. “Skilled Human Welder Intelligence Modeling and Control: Part II-Analysis and Control Applications,” Welding Journal, 93(5): 162s-170s.

Equ quipment pment - Virtu tualize alized d We Welding ing System tem

Illustration of virtualized welding operation. Developed virtualized welding system.

In virtual station a human welder can view the mock up and moves the virtual welding torch accordingly as if he/she is right in front of the work-piece; In welding station a robot arm (Universal Robot UR-5 with six Degree of Freedom) equipped with the welding torch receives commands via Ethernet and performs GTAW.

• Virtual Station and Welding Station

Virtual welding torch.

Equ quipment pment - Virtu tualize alized d We Welding ing System tem

Detailed view of the virtual station. 3D weld pool sensing system.

• A low power laser (19 by 19 structure light

pattern) is projected to the weld pool surface;

• Its reflection from the specular weld pool

surface is intercepted and imaged by Camera 1;

• Weld pool image captured by Camera 2 (with or

without virtual reality (VR) enhancement)

• Major components in virtual station include a

Leap motion tracking sensor, a mock up pipe, a computer screen, and a projector.

3D scanning system Visualization

system

Motion sensor Camera Projector Mockup Screen Detailed view of the visualization result.

Lea earning rning Experiments eriments for

• r Res

esponse ponse Data ta

Teleoperation training experiments

• The welder observe the weld pool under

random welding current, and move the virtual welding torch accordingly.

200 400 600 800 1000 1200 1400 1600 1800 2000 2 4 6 8 10 12 Sample Number Input and Output Speed*3 (mm/s) Current/5 (A) Speedf*3 (mm/s) Width (mm) Length (mm) Convexity*10 (mm)

Measured welding current, weld pool characteristic parameters and human arm movement speed in thirteen training experiments.

Rating ting of

• f Per

erfor formance mance and d Res esponse ponse Data ta

Welder Rating System

• To better distill the correct response of the human welder, the human

welder evaluates the measured data and corresponding back-side weld penetration and assigns a rating (from 0 to 10) in each 5 s interval (offline, thus less skill demanding).

200 400 600 800 1000 1200 1400 1600 1800 2000 5 10 Sample Number Rating (0-10)

Rating assigned by human welder in thirteen dynamic experiments.

Y.K. Liu, Y.M. Zhang, “Iterative local ANFIS based human welder intelligence modeling and control in pipe GTAW process: A data-driven approach,” IEEE/ASME Transactions on Mechatronics, DOI: 10.1109/TMECH.2014.2363050.

ANFI FIS S Ba Based ed Aut utoma

• mate

ted d Rating ting of

• f Data

ta Qua ualit lity y

• Welder Rating System
• Both linear and ANFIS models are proposed to automate the

rating

Linear and ANFIS modeling of the rating (Welder Rating System)

200 400 600 800 1000 1200 1400 1600 1800 2000 2 4 6 8 10 12 Sample Number Human Welder Rating Human Welder Rating Linear Estimated Rating ANFIS Estimated Rating

 

( ) ( ), ( ), ( ), ( ) R k f W k L k C k S k 

Sup uper ervi vised sed Mo Modeling eling

• Good Response data (with ratings greater than 8) are used to model how

human welder adjusts the speed per weld pool surface.

100 200 300 400 500 600 700 0.6 0.8 1 1.2 Sample Number Speedf (mm/s) Measured Speedf Linear Estimated Speedf ANFIS Estimated Speedf

Linear and supervised ANFIS modeling of the welder response.

3D Oper eratio ation n Lear earning ning and nd Rob

• bot
• t

Implementat lementation ion of

• f Hum

uman an Res esponse ponse

• Experiment 1: Different Welding Current
• Experiment 2: Speed Disturbance

40 50 60 70 80 90 100 110 1 2 3 4 5 6 7 8 9 Time (s) Welding Current and Pool Parameters Current/10 (A) Width (mm) Length (mm) Convexity*10 (mm) 40 50 60 70 80 90 100 110

• 0.5

0.5 1 Time (s) Control Inputs Speed (mm/s) Z Adjustment (mm) RX Adjustment/10 (deg) RY Adjustment/10 (deg) 50 60 70 80 90 100 110 2 4 6 8 10 Time (s) Welding Current and Pool Parameters Current/10 (A) Width (mm) Length (mm) Convexity*10 (mm) 50 60 70 80 90 100 110

• 0.5

0.5 1 Time (s) Control Inputs Speed (mm/s) Z Adjustment (mm) RX Adjustment/10 (deg) RY Adjustment/10 (deg)

6. . Fu Futu ture e Direct ections ions

• Varying Gap  feedforward + feedback
• Operation inconsistence: automatic rating of operation quality
• Better Free Demonstration of Human Skills IMU sensor on torch, co-view

helmet

• What is an Co-View Helmet?
• Welder Operation Documentation: Heat Input/Welding Speed/Torch

Orientation/O’clock Position, Weld Pool Size and Shape

• Welder Operation Modeling and Diagnosis
• Improvement from Comparison with Skilled Welder Model
• Program Welding Robot for Intelligent Control
• Automatic Welding Parameters Adjustment Based on Speed and Torch

Orientation/Angle

Ke Key: Mo Mobile le Sen ensor

• r
• Projective Torch : grid

pattern and dot matrix pattern attached, as well as one IMU

• Shield glass (simulated by a

piece of glass covered by a paper)

• Sensory Helmet (camera

tripod to simulate the head movement in 6 DOF)

• Weld pool (Convex spherical

mirror with known geometry) W.J. Zhang, J. Xiao, Y.M. Zhang, 2016. “A mobile sensing system for three-dimensional weld pool measurement in manual GTAW process," Measurement Science and Technology, 27 (2016) 045102 (24pp), doi:10.1088/0957-0233/27/4/045102.

Experimental system configuration Simmer Wireless Inertia Measurement Unit includes: 1. tri-axial accelerometer (Freescale MMA7260Q) 2. tri-axial gyro sensor (InvenSense 500 series) 3. a magnetometer 4. a microprocessor (MSP430F1611) 5. a Bluetooth unit.

Ke Key Com

• mponent

ponent - Iner ertial tial Me Measurement urement Unit it (IMU) MU)

W.J. Zhang, J. Xiao, Y.M. Zhang, 2014. “Navigation of welding torch for arc welding process,” Preprints of the 19th World Congress of The International Federation of Automatic Control, pp. 7158-7163, Cape Town, South Africa, August 24- 29, 2014.

Position experiment The torch is smoothly moved along the 3-D since curve.

The results of torch trajectory position estimation Measurement errors in Position Experiment 2

Tracking Accuracy Verification

W.J. Zhang, J. Xiao, Y.M. Zhang, 2014. “Navigation of welding torch for arc welding process,” Preprints of the 19th World Congress of The International Federation of Automatic Control, pp. 7158-7163, Cape Town, South Africa, August 24-29, 2014.