== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 10:53:46 (1788447226) start mds service on oleg252-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg252-server' oleg252-server: oleg252-server.virtnet: executing load_modules_local oleg252-server: Loading modules from /home/green/git/lustre-release/lustre oleg252-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg252-server: detected 4 online CPUs by sysfs oleg252-server: MODOPTS_LIBCFS= oleg252-server: Force libcfs to create 2 CPU partitions oleg252-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg252-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg252-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg252-server: oleg252-server.virtnet: executing set_default_debug -1 all pdsh@oleg252-client: oleg252-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg252-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg252-server: oleg252-server.virtnet: executing set_default_debug -1 all pdsh@oleg252-client: oleg252-server: ssh exited with exit code 1 Started lustre-OST0000 oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg252-client.virtnet: -o user_xattr,flock 192.168.202.152@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 203 (time 1788447301.00 total 10.01 last 20.29) - open/close 488 (time 1788447311.93 total 20.93 last 26.08) - open/close 776 (time 1788447322.03 total 31.04 last 28.50) - open/close 970 (time 1788447332.73 total 41.73 last 18.14) - open/close 1220 (time 1788447343.09 total 52.10 last 24.12) - open/close 1439 (time 1788447354.82 total 63.82 last 18.68) - open/close 1544 (time 1788447364.83 total 73.83 last 10.49) - open/close 1730 (time 1788447375.53 total 84.54 last 17.37) - open/close 1860 (time 1788447386.02 total 95.02 last 12.39) - open/close 1984 (time 1788447396.92 total 105.93 last 11.38) - open/close 2235 (time 1788447407.42 total 116.43 last 23.90) - open/close 2393 (time 1788447417.84 total 126.85 last 15.16) - open/close 2562 (time 1788447427.86 total 136.86 last 16.88) - open/close 2822 (time 1788447437.86 total 146.86 last 25.99) - open/close 3044 (time 1788447448.07 total 157.07 last 21.75) - open/close 3246 (time 1788447458.21 total 167.21 last 19.92) - open/close 3512 (time 1788447468.39 total 177.40 last 26.12) - open/close 3748 (time 1788447478.63 total 187.63 last 23.06) - open/close 3991 (time 1788447488.63 total 197.64 last 24.29) - open/close 4181 (time 1788447499.07 total 208.08 last 18.20) - open/close 4318 (time 1788447509.98 total 218.98 last 12.56) - open/close 4540 (time 1788447519.99 total 228.99 last 22.18) - open/close 4742 (time 1788447529.99 total 238.99 last 20.20) - open/close 4993 (time 1788447540.47 total 249.47 last 23.96) - open/close 5193 (time 1788447551.21 total 260.21 last 18.62) - open/close 5295 (time 1788447561.23 total 270.23 last 10.17) - open/close 5508 (time 1788447571.71 total 280.71 last 20.33) - open/close 5684 (time 1788447582.19 total 291.19 last 16.79) - open/close 5928 (time 1788447592.19 total 301.20 last 24.39) - open/close 6078 (time 1788447602.66 total 311.67 last 14.33) - open/close 6177 (time 1788447612.82 total 321.83 last 9.74) - open/close 6355 (time 1788447622.87 total 331.87 last 17.72) - open/close 6501 (time 1788447632.89 total 341.89 last 14.57) - open/close 6697 (time 1788447644.11 total 353.11 last 17.47) - open/close 6884 (time 1788447654.11 total 363.11 last 18.70) - open/close 7028 (time 1788447664.18 total 373.18 last 14.30) - open/close 7162 (time 1788447674.18 total 383.18 last 13.40) - open/close 7295 (time 1788447684.99 total 394.00 last 12.30) - open/close 7471 (time 1788447695.00 total 404.01 last 17.58) - open/close 7752 (time 1788447705.00 total 414.01 last 28.10) - open/close 7971 (time 1788447715.01 total 424.01 last 21.90) - open/close 8152 (time 1788447725.50 total 434.50 last 17.25) - open/close 8385 (time 1788447736.11 total 445.11 last 21.96) - open/close 8442 (time 1788447748.21 total 457.21 last 4.71) - open/close 8657 (time 1788447758.22 total 467.23 last 21.47) - open/close 8897 (time 1788447768.48 total 477.48 last 23.40) - open/close 9145 (time 1788447779.14 total 488.15 last 23.26) - open/close 9379 (time 1788447789.67 total 498.68 last 22.22) - open/close 9602 (time 1788447799.68 total 508.69 last 22.28) - open/close 9827 (time 1788447810.47 total 519.47 last 20.86) - open/close 10000 (time 1788447818.87 total 527.88 last 20.59) - open/close 10132 (time 1788447830.89 total 539.89 last 10.99) - open/close 10252 (time 1788447841.34 total 550.35 last 11.48) - open/close 10364 (time 1788447851.73 total 560.74 last 10.78) - open/close 10423 (time 1788447861.74 total 570.75 last 5.90) - open/close 10523 (time 1788447872.57 total 581.57 last 9.23) - open/close 10686 (time 1788447883.46 total 592.46 last 14.97) - open/close 10867 (time 1788447894.40 total 603.41 last 16.54) - open/close 11082 (time 1788447904.86 total 613.87 last 20.56) - open/close 11346 (time 1788447914.88 total 623.88 last 26.36) - open/close 11553 (time 1788447925.37 total 634.37 last 19.73) - open/close 11729 (time 1788447935.92 total 644.92 last 16.69) - open/close 11879 (time 1788447948.85 total 657.86 last 11.59) - open/close 12124 (time 1788447958.85 total 667.86 last 24.50) - open/close 12296 (time 1788447969.27 total 678.27 last 16.52) - open/close 12361 (time 1788447979.77 total 688.78 last 6.19) - open/close 12584 (time 1788447990.73 total 699.73 last 20.35) - open/close 12639 (time 1788448000.82 total 709.82 last 5.45) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-12785) error: No space left on device total: 12785 open/close in 718.75 seconds: 17.79 ops/second /home/green/git/lustre-release/lustre/tests/conf-sanity.sh: line 9017: kill: (76053) - No such process - unlinked 0 (time 1788448014 ; total 0 ; last 0) - unlinked 10000 (time 1788448105 ; total 91 ; last 91) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-12785) error: No such file or directory total: 12785 unlinks in 112 seconds: 114.151787 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg252-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg252-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg252-server stop mds service on oleg252-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg252-server unloading modules via unload_modules_local on: 'oleg252-server' oleg252-server: oleg252-server.virtnet: executing unload_modules_local oleg252-server: modules unloaded. pdsh@oleg252-client: oleg252-server: ssh exited with exit code 2 pdsh@oleg252-client: oleg252-client: ssh exited with exit code 2 pdsh@oleg252-client: oleg252-client: ssh exited with exit code 2 pdsh@oleg252-client: oleg252-server: ssh exited with exit code 2