== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 12:57:44 (1772474264) start mds service on oleg105-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg105-server' oleg105-server: oleg105-server.virtnet: executing load_modules_local oleg105-server: Loading modules from /home/green/git/lustre-release/lustre oleg105-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg105-server: detected 4 online CPUs by sysfs oleg105-server: MODOPTS_LIBCFS= oleg105-server: Force libcfs to create 2 CPU partitions oleg105-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg105-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg105-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg105-server: oleg105-server.virtnet: executing set_default_debug -1 all pdsh@oleg105-client: oleg105-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg105-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg105-server: oleg105-server.virtnet: executing set_default_debug -1 all pdsh@oleg105-client: oleg105-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg105-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg105-server: oleg105-server.virtnet: executing set_default_debug -1 all pdsh@oleg105-client: oleg105-server: ssh exited with exit code 1 Started lustre-OST0000 oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg105-client.virtnet: -o user_xattr,flock 192.168.201.105@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 290 (time 1772474320.43 total 10.46 last 27.71) - open/close 591 (time 1772474330.92 total 20.95 last 28.69) - open/close 1006 (time 1772474341.93 total 31.96 last 37.70) - open/close 1248 (time 1772474351.98 total 42.01 last 24.08) - open/close 1468 (time 1772474362.95 total 52.98 last 20.06) - open/close 1743 (time 1772474373.28 total 63.32 last 26.60) - open/close 1963 (time 1772474384.05 total 74.08 last 20.44) - open/close 2187 (time 1772474394.08 total 84.12 last 22.32) - open/close 2424 (time 1772474404.90 total 94.94 last 21.90) - open/close 2602 (time 1772474415.01 total 105.05 last 17.61) - open/close 3155 (time 1772474425.02 total 115.05 last 55.26) - open/close 3681 (time 1772474435.04 total 125.08 last 52.45) - open/close 3874 (time 1772474445.16 total 135.19 last 19.09) - open/close 4130 (time 1772474455.46 total 145.50 last 24.84) - open/close 4620 (time 1772474465.57 total 155.61 last 48.45) - open/close 5087 (time 1772474475.57 total 165.61 last 46.69) - open/close 5771 (time 1772474485.93 total 175.96 last 66.07) - open/close 6400 (time 1772474495.93 total 185.96 last 62.89) - open/close 7021 (time 1772474506.08 total 196.12 last 61.15) - open/close 7665 (time 1772474516.39 total 206.43 last 62.49) - open/close 7984 (time 1772474526.95 total 216.99 last 30.21) - open/close 8544 (time 1772474537.32 total 227.35 last 54.01) - open/close 9095 (time 1772474547.75 total 237.79 last 52.80) - open/close 9701 (time 1772474558.06 total 248.10 last 58.78) - open/close 9880 (time 1772474568.36 total 258.40 last 17.39) - open/close 10000 (time 1772474570.90 total 260.94 last 47.19) - open/close 10377 (time 1772474581.92 total 271.96 last 34.21) - open/close 10955 (time 1772474592.23 total 282.26 last 56.10) - open/close 11628 (time 1772474602.53 total 292.57 last 65.32) - open/close 12213 (time 1772474612.78 total 302.81 last 57.10) - open/close 12744 (time 1772474623.85 total 313.89 last 47.95) - open/close 13123 (time 1772474634.60 total 324.64 last 35.25) - open/close 13464 (time 1772474645.49 total 335.52 last 31.32) - open/close 13976 (time 1772474655.53 total 345.56 last 51.00) - open/close 14957 (time 1772474665.53 total 355.56 last 98.08) - open/close 17354 (time 1772474675.53 total 365.57 last 239.63) - open/close 19513 (time 1772474685.53 total 375.57 last 215.83) - open/close 21968 (time 1772474695.54 total 385.57 last 245.44) - open/close 25249 (time 1772474705.54 total 395.58 last 328.00) - open/close 29272 (time 1772474715.54 total 405.58 last 402.26) - open/close 32261 (time 1772474725.54 total 415.58 last 298.85) - open/close 35253 (time 1772474735.54 total 425.58 last 299.17) - open/close 38783 (time 1772474745.54 total 435.58 last 352.99) - open/close 40000 (time 1772474749.42 total 439.45 last 314.24) - open/close 43253 (time 1772474759.42 total 449.46 last 325.11) - open/close 46994 (time 1772474769.42 total 459.46 last 374.05) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 466.18 seconds: 106.46 ops/second - unlinked 0 (time 1772474777 ; total 0 ; last 0) - unlinked 10000 (time 1772474794 ; total 17 ; last 17) - unlinked 20000 (time 1772474810 ; total 33 ; last 16) - unlinked 30000 (time 1772474825 ; total 48 ; last 15) - unlinked 40000 (time 1772474838 ; total 61 ; last 13) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 75 seconds: 661.760010 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg105-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg105-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg105-server stop mds service on oleg105-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg105-server stop mds service on oleg105-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg105-server unloading modules via unload_modules_local on: 'oleg105-server' oleg105-server: oleg105-server.virtnet: executing unload_modules_local oleg105-server: modules unloaded. pdsh@oleg105-client: oleg105-client: ssh exited with exit code 2 pdsh@oleg105-client: oleg105-server: ssh exited with exit code 2 pdsh@oleg105-client: oleg105-server: ssh exited with exit code 2 pdsh@oleg105-client: oleg105-client: ssh exited with exit code 2