== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 12:56:51 (1772474211) start mds service on oleg449-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg449-server' oleg449-server: oleg449-server.virtnet: executing load_modules_local oleg449-server: Loading modules from /home/green/git/lustre-release/lustre oleg449-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg449-server: detected 4 online CPUs by sysfs oleg449-server: MODOPTS_LIBCFS= oleg449-server: Force libcfs to create 2 CPU partitions oleg449-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg449-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg449-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg449-server: oleg449-server.virtnet: executing set_default_debug -1 all pdsh@oleg449-client: oleg449-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg449-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg449-server: oleg449-server.virtnet: executing set_default_debug -1 all pdsh@oleg449-client: oleg449-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg449-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg449-server: oleg449-server.virtnet: executing set_default_debug -1 all pdsh@oleg449-client: oleg449-server: ssh exited with exit code 1 Started lustre-OST0000 oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg449-client.virtnet: -o user_xattr,flock 192.168.204.149@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 250 (time 1772474279.33 total 10.01 last 24.97) - open/close 358 (time 1772474289.95 total 20.63 last 10.17) - open/close 523 (time 1772474300.46 total 31.14 last 15.71) - open/close 595 (time 1772474311.72 total 42.39 last 6.40) - open/close 773 (time 1772474322.09 total 52.77 last 17.16) - open/close 826 (time 1772474332.10 total 62.77 last 5.30) - open/close 953 (time 1772474343.07 total 73.75 last 11.57) - open/close 1185 (time 1772474353.26 total 83.94 last 22.79) - open/close 1287 (time 1772474364.01 total 94.69 last 9.48) - open/close 1421 (time 1772474374.95 total 105.63 last 12.25) - open/close 1603 (time 1772474385.76 total 116.44 last 16.84) - open/close 1742 (time 1772474396.51 total 127.19 last 12.93) - open/close 1824 (time 1772474406.82 total 137.50 last 7.95) - open/close 1943 (time 1772474418.40 total 149.08 last 10.27) - open/close 2239 (time 1772474429.40 total 160.08 last 26.90) - open/close 2361 (time 1772474440.09 total 170.77 last 11.42) - open/close 2682 (time 1772474451.11 total 181.79 last 29.14) - open/close 2902 (time 1772474461.28 total 191.96 last 21.62) - open/close 3115 (time 1772474472.10 total 202.78 last 19.69) - open/close 3352 (time 1772474482.85 total 213.53 last 22.04) - open/close 3447 (time 1772474494.44 total 225.12 last 8.20) - open/close 3508 (time 1772474504.44 total 235.12 last 6.10) - open/close 3741 (time 1772474516.77 total 247.45 last 18.89) - open/close 4053 (time 1772474526.78 total 257.46 last 31.18) - open/close 4218 (time 1772474536.99 total 267.67 last 16.16) - open/close 4459 (time 1772474547.18 total 277.86 last 23.66) - open/close 4754 (time 1772474557.47 total 288.15 last 28.67) - open/close 5042 (time 1772474567.71 total 298.39 last 28.12) - open/close 5264 (time 1772474578.60 total 309.28 last 20.38) - open/close 5549 (time 1772474589.16 total 319.84 last 26.98) - open/close 5826 (time 1772474599.78 total 330.45 last 26.10) - open/close 6198 (time 1772474610.09 total 340.76 last 36.08) - open/close 6500 (time 1772474620.26 total 350.93 last 29.70) - open/close 6758 (time 1772474630.75 total 361.43 last 24.58) - open/close 6955 (time 1772474641.69 total 372.37 last 18.01) - open/close 7218 (time 1772474651.82 total 382.50 last 25.98) - open/close 7517 (time 1772474661.82 total 392.50 last 29.89) - open/close 7851 (time 1772474672.80 total 403.48 last 30.41) - open/close 8162 (time 1772474683.61 total 414.29 last 28.77) - open/close 8370 (time 1772474693.73 total 424.41 last 20.57) - open/close 8716 (time 1772474703.73 total 434.41 last 34.59) - open/close 9039 (time 1772474713.74 total 444.42 last 32.27) - open/close 9228 (time 1772474723.74 total 454.42 last 18.90) - open/close 9445 (time 1772474733.98 total 464.66 last 21.19) - open/close 9736 (time 1772474744.23 total 474.91 last 28.38) - open/close 10000 (time 1772474752.66 total 483.34 last 31.31) - open/close 10215 (time 1772474762.75 total 493.43 last 21.31) - open/close 10405 (time 1772474772.77 total 503.45 last 18.96) - open/close 11130 (time 1772474782.78 total 513.45 last 72.48) - open/close 12515 (time 1772474792.78 total 523.45 last 138.49) - open/close 13906 (time 1772474802.78 total 533.46 last 139.02) - open/close 15260 (time 1772474812.79 total 543.47 last 135.24) - open/close 16610 (time 1772474822.79 total 553.47 last 134.99) - open/close 17830 (time 1772474832.80 total 563.48 last 121.92) - open/close 19078 (time 1772474842.80 total 573.48 last 124.77) - open/close 20000 (time 1772474849.54 total 580.22 last 136.88) - open/close 21428 (time 1772474859.55 total 590.23 last 142.62) - open/close 22811 (time 1772474869.55 total 600.23 last 138.29) - open/close 24210 (time 1772474879.55 total 610.23 last 139.87) - open/close 25682 (time 1772474889.56 total 620.24 last 147.15) - open/close 27094 (time 1772474899.56 total 630.24 last 141.20) - open/close 28504 (time 1772474909.57 total 640.25 last 140.80) - open/close 29970 (time 1772474919.58 total 650.26 last 146.50) - open/close 31336 (time 1772474929.58 total 660.26 last 136.57) - open/close 32791 (time 1772474939.59 total 670.27 last 145.38) - open/close 34327 (time 1772474949.61 total 680.29 last 153.31) - open/close 35998 (time 1772474959.61 total 690.29 last 167.03) - open/close 37471 (time 1772474969.62 total 700.30 last 147.21) - open/close 38820 (time 1772474979.62 total 710.30 last 134.89) - open/close 40000 (time 1772474986.78 total 717.46 last 164.74) - open/close 41539 (time 1772474996.79 total 727.47 last 153.82) - open/close 42996 (time 1772475006.79 total 737.47 last 145.68) - open/close 44362 (time 1772475016.79 total 747.47 last 136.60) - open/close 45769 (time 1772475026.79 total 757.47 last 140.66) - open/close 47486 (time 1772475036.80 total 767.48 last 171.59) - open/close 49322 (time 1772475046.80 total 777.48 last 183.54) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 779.34 seconds: 63.68 ops/second - unlinked 0 (time 1772475050 ; total 0 ; last 0) - unlinked 10000 (time 1772475091 ; total 41 ; last 41) - unlinked 20000 (time 1772475126 ; total 76 ; last 35) - unlinked 30000 (time 1772475160 ; total 110 ; last 34) - unlinked 40000 (time 1772475192 ; total 142 ; last 32) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 167 seconds: 297.197601 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg449-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg449-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg449-server stop mds service on oleg449-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg449-server stop mds service on oleg449-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg449-server unloading modules via unload_modules_local on: 'oleg449-server' oleg449-server: oleg449-server.virtnet: executing unload_modules_local oleg449-server: modules unloaded. pdsh@oleg449-client: oleg449-client: ssh exited with exit code 2 pdsh@oleg449-client: oleg449-server: ssh exited with exit code 2 pdsh@oleg449-client: oleg449-server: ssh exited with exit code 2 pdsh@oleg449-client: oleg449-client: ssh exited with exit code 2