jive5ab 2.8.1 and support acts m5copy and vbs_{fs,ls,rm} May 2017 - - PowerPoint PPT Presentation

▶
jive5ab 2 8 1
SMART_READER_LITE
LIVE PREVIEW

jive5ab 2.8.1 and support acts m5copy and vbs_{fs,ls,rm} May 2017 - - PowerPoint PPT Presentation

Sep 18, 2022 283 likes •576 views

jive5ab 2.8.1 and support acts m5copy and vbs_{fs,ls,rm} May 2017 TOG - Ventspils verkouter@jive.eu New developments in jive5ab 2.8.1 May 2017 TOG - Ventspils verkouter@jive.eu Many fjxes in 2.8.1 SIGSEGV reported by one (1) FlexBuff

slide-1
SLIDE 1

TOG - Ventspils May 2017 verkouter@jive.eu

jive5ab 2.8.1

and support acts m5copy and vbs_{fs,ls,rm}

slide-2
SLIDE 2

TOG - Ventspils May 2017 verkouter@jive.eu

New developments in jive5ab 2.8.1

slide-3
SLIDE 3

TOG - Ventspils May 2017 verkouter@jive.eu

Many fjxes in 2.8.1

SIGSEGV reported by one (1) FlexBuff station: showstopper! O/S and/or compiler and/or hardware sensitive? sure it was memory corruption related to multithreading issue debugging fun:

  • disk2file (on FlexBuff) seemed to be trigger
  • did not trigger @JIVE
  • observed different malfunction after several hours of

hammering (output to screen stopped but no SIGSEGV)

slide-4
SLIDE 4

TOG - Ventspils May 2017 verkouter@jive.eu

Many fjxes in 2.8.1

SIGSEGV reported by one (1) FlexBuff station: showstopper! Did not fix the problem at all ... Now what?

  • assess MT* safety of all systemcalls
  • a number of MT-unsafe versions were used (FIXED)
  • some MT-safe wrappers needed to be written
  • this is all invisible to the user/outside

(*) MT-safe = Multi-Thread safe = reentrant:

  • cf. POSIX1.c: a "function whose effect, when called by two or more threads, is guaranteed to be as

if the threads each executed the function one after another in an undefined order, even if the actual execution is interleaved”

slide-5
SLIDE 5

TOG - Ventspils May 2017 verkouter@jive.eu

Many fjxes in 2.8.1

SIGSEGV reported by one (1) FlexBuff station: showstopper! Did not fix the problem at all ... Now what?

  • threads closing file descriptors in consistent manner (FIXED)
  • serialize access to memory deallocator (FIXED)
  • again totally invisible for user/outside
slide-6
SLIDE 6

TOG - Ventspils May 2017 verkouter@jive.eu

Many fjxes in 2.8.1

SIGSEGV reported by one (1) FlexBuff station: showstopper! Led to finding systematic problem: Now what?

  • actually think about what happens when triggered @station

inadvertently using memory after it was freed in disk2file

slide-7
SLIDE 7

TOG - Ventspils May 2017 verkouter@jive.eu

Many fjxes in 2.8.1

SIGSEGV reported by one (1) FlexBuff station: showstopper! Adding insult to injury: Now what?

  • FIXED!

introduced by self in 2.8.0 in attempt to eliminate memory leaks

slide-8
SLIDE 8

TOG - Ventspils May 2017 verkouter@jive.eu

Many fjxes in 2.8.1

SIGSEGV reported by one (1) FlexBuff station: showstopper! All fixed; code in (even) better MT-safe shape now Found same internal approach in

  • disk2net (FlexBuff)
  • file2disk
slide-9
SLIDE 9

TOG - Ventspils May 2017 verkouter@jive.eu

More fjxes in 2.8.1

Outward visible

  • record? reply on FlexBuff made FS unhappy
  • introduced by self in (failed ...) attempt to consistentify the

reply format across platforms (FIXED)

  • suggestion by BeppeM record? (FlexBuff) returns
  • record? 0 : [on|off] : ...
  • was active|inactive
slide-10
SLIDE 10

TOG - Ventspils May 2017 verkouter@jive.eu

More fjxes in 2.8.1

Outward visible

  • jive5ab 2.8.0 lost ability to erase/initialize new disk packs
  • introduced by self, focussing on non-bank mode ...
  • leftovers from change-of-internal-unit of frequency MHz→Hz
  • few instances overlooked in high-res time stamps of 2.8.0
slide-11
SLIDE 11

TOG - Ventspils May 2017 verkouter@jive.eu

More fjxes in 2.8.1

Outward visible

  • net2out cleanup code was not always called ....
  • could get stuck in mode w/o being able to repair
  • (unlikely) failure in record=on on Mark5B could trigger ...
  • could get stuck in mode w/o being able to repair
slide-12
SLIDE 12

TOG - Ventspils May 2017 verkouter@jive.eu

2.8.1 and the past

Everyone upgraded?

  • provide only 64-bit WHEEZY binary package?
  • with- (for Mark5*), and
  • without SDK9.4 (FlexBuff)
slide-13
SLIDE 13

TOG - Ventspils May 2017 verkouter@jive.eu

m5copy from 1.48→1.53

Fixes

  • handle changes in record? reply
  • file2net? can return mixed state of running+not connected
  • causes m5copy to terminate w/ error
  • wildcarded files are now transferred in alphabetical order
  • sped up scanning wildcarded vbs recordings
  • attempt to detect copying scan to bank label or VSN
  • missing 2nd slash in mk5://.../A or mk5://.../MPI-001
  • should read mk5://.../A/ or mk5://.../MPI-001/
slide-14
SLIDE 14

TOG - Ventspils May 2017 verkouter@jive.eu

vbs_fs

Fixes

  • file descriptor leak indexing Mark6 recordings
  • detect random or sequential access pattern
  • only read whole chunk if sequential access detected
  • add –I <pattern> command line option
  • index only recordings matching <pattern>
slide-15
SLIDE 15

TOG - Ventspils May 2017 verkouter@jive.eu

All tools: jive5ab, m5copy, vbs_*

Fix in all utilities:

  • now able to deal with scan names with “.” or “+” in the name
slide-16
SLIDE 16

TOG - Ventspils May 2017 verkouter@jive.eu

# !

slide-17
SLIDE 17

TOG - Ventspils May 2017 verkouter@jive.eu

# ! / bi n/ sh l s *. t xt | sed . . .

slide-18
SLIDE 18

TOG - Ventspils May 2017 verkouter@jive.eu

# ! / usr / bi n/ pyt hon i m por t r e r xVSN = r e. com pi l e(

slide-19
SLIDE 19

TOG - Ventspils May 2017 verkouter@jive.eu

# ! . . . / m 5copy < . . . > gg035a_ys_no0001 n16c1_i b_no0120 . . .

slide-20
SLIDE 20

TOG - Ventspils May 2017 verkouter@jive.eu

# ! . . . / m 5copy < . . . > gg035a_ys_no0001 n16c1_i b_no0120 . . .

slide-21
SLIDE 21

TOG - Ventspils May 2017 verkouter@jive.eu

# ! . . . / m 5copy < . . . > gg035a_ys_no0001 n16c1_i b_no0120 . . .

‘template’ command line

slide-22
SLIDE 22

TOG - Ventspils May 2017 verkouter@jive.eu

# ! . . . / m 5copy –udt –m 4500 {0} fi l e: / / . . /

slide-23
SLIDE 23

TOG - Ventspils May 2017 verkouter@jive.eu

# ! . . . / m 5copy –udt –m 4500 {0} fi l e: / / . . /

slide-24
SLIDE 24

TOG - Ventspils May 2017 verkouter@jive.eu

# ! . . . / m 5copy –udt –m 4500 {0} fi l e: / / . . /

Python string formatting: “replace with field #n” – see URL below

https://docs.python.org/2/library/string.html#format-specification-mini-language

slide-25
SLIDE 25

TOG - Ventspils May 2017 verkouter@jive.eu

# ! . . . / m 5copy –udt –m 4500 {0} fi l e: / / . . /

Not limited to one field per input line No spaces in fields though 

https://docs.python.org/2/library/string.html#format-specification-mini-language

slide-26
SLIDE 26

TOG - Ventspils May 2017 verkouter@jive.eu

# ! . . . / m 5copy –udt –r {0} m k5: / / . . / {1} {2} # send scans 1- 10 t

  • Bonn

@ 128M bps 128M 1- 10 fi l e: / / i

  • 03.

bonn. m pg. de/ dat a/ # send n16c1 t

  • J

I VE @ 2G bps 2G n16c1_hh_* vbs: / / fl exbuf 6. j i ve. nl /

slide-27
SLIDE 27

TOG - Ventspils May 2017 verkouter@jive.eu

# ! . . . / m 5copy –udt –r {0} m k5: / / . . / {1} {2} # send scans 1- 10 t

  • Bonn

@ 128M bps 128M 1- 10 fi l e: / / i

  • 03.

bonn. m pg. de/ dat a/ # send n16c1 t

  • J

I VE @ 2G bps 2G n16c1_hh_* vbs: / / fl exbuf 6. j i ve. nl /

slide-28
SLIDE 28

TOG - Ventspils May 2017 verkouter@jive.eu

# ! . . . / m 5copy –udt –r {0} m k5: / / . . / {1} {2} # send scans 1- 10 t

  • Bonn

@ 128M bps 128M 1- 10 fi l e: / / i

  • 03.

bonn. m pg. de/ dat a/ # send n16c1 t

  • J

I VE @ 2G bps 2G n16c1_hh_* vbs: / / fl exbuf 6. j i ve. nl /