== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 05:58:07 (1765105087) start mds service on oleg424-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg424-server' oleg424-server: oleg424-server.virtnet: executing load_modules_local oleg424-server: Loading modules from /home/green/git/lustre-release/lustre oleg424-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg424-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg424-server: detected 4 online CPUs by sysfs oleg424-server: MODOPTS_LIBCFS= oleg424-server: Force libcfs to create 2 CPU partitions oleg424-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg424-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg424-server: oleg424-server.virtnet: executing set_default_debug -1 all pdsh@oleg424-client: oleg424-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg424-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg424-server: oleg424-server.virtnet: executing set_default_debug -1 all pdsh@oleg424-client: oleg424-server: ssh exited with exit code 1 Started lustre-OST0000 oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg424-client.virtnet: -o user_xattr,flock 192.168.204.124@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 208 (time 1765105134.35 total 10.36 last 20.08) - open/close 420 (time 1765105144.79 total 20.79 last 20.31) - open/close 753 (time 1765105154.83 total 30.84 last 33.15) - open/close 998 (time 1765105165.33 total 41.34 last 23.33) - open/close 1151 (time 1765105176.34 total 52.35 last 13.90) - open/close 1371 (time 1765105186.77 total 62.78 last 21.09) - open/close 1568 (time 1765105197.01 total 73.02 last 19.24) - open/close 1690 (time 1765105207.83 total 83.84 last 11.27) - open/close 1801 (time 1765105217.88 total 93.88 last 11.05) - open/close 1981 (time 1765105228.19 total 104.20 last 17.45) - open/close 2227 (time 1765105238.74 total 114.74 last 23.33) - open/close 2473 (time 1765105249.10 total 125.11 last 23.73) - open/close 2776 (time 1765105259.42 total 135.43 last 29.36) - open/close 2964 (time 1765105269.52 total 145.53 last 18.62) - open/close 3127 (time 1765105279.53 total 155.54 last 16.28) - open/close 3405 (time 1765105290.00 total 166.01 last 26.55) - open/close 3752 (time 1765105300.43 total 176.44 last 33.26) - open/close 3925 (time 1765105310.55 total 186.55 last 17.11) - open/close 4102 (time 1765105320.55 total 196.56 last 17.69) - open/close 4312 (time 1765105330.78 total 206.78 last 20.54) - open/close 4510 (time 1765105340.89 total 216.90 last 19.58) - open/close 4653 (time 1765105351.56 total 227.57 last 13.40) - open/close 4963 (time 1765105362.51 total 238.51 last 28.33) - open/close 5278 (time 1765105372.51 total 248.52 last 31.49) - open/close 5537 (time 1765105383.27 total 259.27 last 24.08) - open/close 5651 (time 1765105393.96 total 269.96 last 10.66) - open/close 5845 (time 1765105403.99 total 279.99 last 19.34) - open/close 6150 (time 1765105414.79 total 290.80 last 28.21) - open/close 6450 (time 1765105424.80 total 300.81 last 29.97) - open/close 6722 (time 1765105434.81 total 310.82 last 27.18) - open/close 6921 (time 1765105445.07 total 321.08 last 19.40) - open/close 7128 (time 1765105455.76 total 331.76 last 19.37) - open/close 7310 (time 1765105465.87 total 341.88 last 17.99) - open/close 7569 (time 1765105476.76 total 352.76 last 23.79) - open/close 7780 (time 1765105487.51 total 363.51 last 19.63) - open/close 7952 (time 1765105497.56 total 373.56 last 17.11) - open/close 8185 (time 1765105507.67 total 383.68 last 23.04) - open/close 8429 (time 1765105517.67 total 393.68 last 24.39) - open/close 8551 (time 1765105528.40 total 404.40 last 11.38) - open/close 8699 (time 1765105539.29 total 415.30 last 13.59) - open/close 8799 (time 1765105549.90 total 425.91 last 9.42) - open/close 8929 (time 1765105559.91 total 435.92 last 12.99) - open/close 9024 (time 1765105570.58 total 446.59 last 8.90) - open/close 9321 (time 1765105580.59 total 456.59 last 29.69) - open/close 9466 (time 1765105590.67 total 466.68 last 14.38) - open/close 9554 (time 1765105601.24 total 477.24 last 8.33) - open/close 9777 (time 1765105611.85 total 487.86 last 21.00) - open/close 9901 (time 1765105622.42 total 498.43 last 11.73) - open/close 10000 (time 1765105625.31 total 501.32 last 34.32) - open/close 10328 (time 1765105635.31 total 511.32 last 32.79) - open/close 11714 (time 1765105645.31 total 521.32 last 138.58) - open/close 13091 (time 1765105655.32 total 531.32 last 137.64) - open/close 14070 (time 1765105665.32 total 541.33 last 97.86) - open/close 15097 (time 1765105675.32 total 551.33 last 102.69) - open/close 16040 (time 1765105685.33 total 561.34 last 94.23) - open/close 16981 (time 1765105695.33 total 571.34 last 94.09) - open/close 17976 (time 1765105705.34 total 581.35 last 99.41) - open/close 18984 (time 1765105715.34 total 591.35 last 100.75) - open/close 20000 (time 1765105724.96 total 600.97 last 105.63) - open/close 21303 (time 1765105734.96 total 610.97 last 130.30) - open/close 22546 (time 1765105744.97 total 620.98 last 124.24) - open/close 23631 (time 1765105754.98 total 630.99 last 108.33) - open/close 24715 (time 1765105764.99 total 641.00 last 108.33) - open/close 25920 (time 1765105775.00 total 651.01 last 120.39) - open/close 27045 (time 1765105785.00 total 661.01 last 112.43) - open/close 28245 (time 1765105795.01 total 671.02 last 119.95) - open/close 29680 (time 1765105805.01 total 681.02 last 143.48) - open/close 30000 (time 1765105807.29 total 683.29 last 140.55) - open/close 31481 (time 1765105817.29 total 693.30 last 147.98) - open/close 32988 (time 1765105827.30 total 703.30 last 150.68) - open/close 34224 (time 1765105837.30 total 713.30 last 123.60) - open/close 35540 (time 1765105847.30 total 723.31 last 131.57) - open/close 37250 (time 1765105857.30 total 733.31 last 170.93) - open/close 38613 (time 1765105867.31 total 743.32 last 136.23) - open/close 39886 (time 1765105877.31 total 753.32 last 127.25) - open/close 41224 (time 1765105887.31 total 763.32 last 133.77) - open/close 42780 (time 1765105897.32 total 773.33 last 155.50) - open/close 44118 (time 1765105907.32 total 783.33 last 133.78) - open/close 45365 (time 1765105917.33 total 793.33 last 124.64) - open/close 46504 (time 1765105927.33 total 803.34 last 113.87) - open/close 47593 (time 1765105937.33 total 813.34 last 108.87) - open/close 48738 (time 1765105947.33 total 823.34 last 114.48) total: 50000 open/close in 832.11 seconds: 60.09 ops/second - unlinked 0 (time 1765105958 ; total 0 ; last 0) - unlinked 10000 (time 1765106008 ; total 50 ; last 50) - unlinked 20000 (time 1765106042 ; total 84 ; last 34) - unlinked 30000 (time 1765106079 ; total 121 ; last 37) - unlinked 40000 (time 1765106120 ; total 162 ; last 41) total: 50000 unlinks in 189 seconds: 264.550262 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg424-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg424-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg424-server stop mds service on oleg424-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg424-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg424-server' oleg424-server: oleg424-server.virtnet: executing unload_modules_local oleg424-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg424-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg424-client: oleg424-client: ssh exited with exit code 2 pdsh@oleg424-client: oleg424-server: ssh exited with exit code 2 pdsh@oleg424-client: oleg424-server: ssh exited with exit code 2 pdsh@oleg424-client: oleg424-client: ssh exited with exit code 2