== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 19:59:55 (1763341195) start mds service on oleg444-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg444-server' oleg444-server: oleg444-server.virtnet: executing load_modules_local oleg444-server: Loading modules from /home/green/git/lustre-release/lustre oleg444-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg444-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg444-server: detected 4 online CPUs by sysfs oleg444-server: MODOPTS_LIBCFS= oleg444-server: Force libcfs to create 2 CPU partitions oleg444-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg444-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg444-server: oleg444-server.virtnet: executing set_default_debug -1 all pdsh@oleg444-client: oleg444-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg444-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg444-server: oleg444-server.virtnet: executing set_default_debug -1 all pdsh@oleg444-client: oleg444-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg444-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg444-server: oleg444-server.virtnet: executing set_default_debug -1 all pdsh@oleg444-client: oleg444-server: ssh exited with exit code 1 Started lustre-OST0000 oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg444-client.virtnet: -o user_xattr,flock 192.168.204.144@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 214 (time 1763341287.41 total 10.01 last 21.38) - open/close 424 (time 1763341297.42 total 20.02 last 20.98) - open/close 640 (time 1763341307.65 total 30.25 last 21.11) - open/close 841 (time 1763341318.11 total 40.70 last 19.23) - open/close 1106 (time 1763341328.13 total 50.72 last 26.46) - open/close 1367 (time 1763341338.16 total 60.76 last 26.01) - open/close 1582 (time 1763341348.24 total 70.83 last 21.34) - open/close 1823 (time 1763341359.00 total 81.59 last 22.40) - open/close 1998 (time 1763341369.49 total 92.09 last 16.67) - open/close 2231 (time 1763341379.51 total 102.10 last 23.26) - open/close 2400 (time 1763341391.84 total 114.44 last 13.70) - open/close 2610 (time 1763341401.86 total 124.45 last 20.96) - open/close 2819 (time 1763341411.87 total 134.46 last 20.88) - open/close 3073 (time 1763341422.55 total 145.14 last 23.78) - open/close 3310 (time 1763341433.16 total 155.75 last 22.34) - open/close 3571 (time 1763341443.16 total 165.76 last 26.09) - open/close 3778 (time 1763341453.17 total 175.77 last 20.68) - open/close 3941 (time 1763341463.93 total 186.52 last 15.16) - open/close 4097 (time 1763341474.49 total 197.09 last 14.76) - open/close 4260 (time 1763341484.51 total 207.11 last 16.27) - open/close 4391 (time 1763341494.53 total 217.13 last 13.07) - open/close 4528 (time 1763341504.56 total 227.15 last 13.66) - open/close 4737 (time 1763341515.17 total 237.76 last 19.70) - open/close 5004 (time 1763341525.19 total 247.79 last 26.63) - open/close 5168 (time 1763341536.22 total 258.82 last 14.87) - open/close 5248 (time 1763341547.32 total 269.91 last 7.21) - open/close 5423 (time 1763341558.13 total 280.73 last 16.18) - open/close 5644 (time 1763341570.51 total 293.10 last 17.86) - open/close 5873 (time 1763341580.76 total 303.36 last 22.34) - open/close 6060 (time 1763341590.76 total 313.36 last 18.69) - open/close 6292 (time 1763341600.77 total 323.37 last 23.18) - open/close 6492 (time 1763341610.89 total 333.49 last 19.77) - open/close 6661 (time 1763341621.36 total 343.95 last 16.15) - open/close 6892 (time 1763341632.35 total 354.94 last 21.02) - open/close 7080 (time 1763341642.36 total 364.96 last 18.77) - open/close 7329 (time 1763341654.99 total 377.59 last 19.72) - open/close 7491 (time 1763341665.81 total 388.41 last 14.97) - open/close 7693 (time 1763341676.19 total 398.78 last 19.47) - open/close 7945 (time 1763341688.30 total 410.90 last 20.80) - open/close 8186 (time 1763341698.33 total 420.92 last 24.03) - open/close 8304 (time 1763341708.34 total 430.94 last 11.79) - open/close 8434 (time 1763341720.28 total 442.87 last 10.89) - open/close 8510 (time 1763341732.24 total 454.83 last 6.36) - open/close 8616 (time 1763341743.12 total 465.71 last 9.74) - open/close 8650 (time 1763341755.42 total 478.01 last 2.76) - open/close 8774 (time 1763341765.44 total 488.04 last 12.37) - open/close 8926 (time 1763341775.98 total 498.57 last 14.43) - open/close 9058 (time 1763341786.23 total 508.82 last 12.88) - open/close 9286 (time 1763341797.24 total 519.84 last 20.70) - open/close 9423 (time 1763341807.85 total 530.44 last 12.92) - open/close 9529 (time 1763341820.39 total 542.98 last 8.46) - open/close 9697 (time 1763341831.05 total 553.64 last 15.76) - open/close 9920 (time 1763341841.35 total 563.95 last 21.64) - open/close 10000 (time 1763341844.82 total 567.42 last 23.05) - open/close 10229 (time 1763341854.87 total 577.46 last 22.80) - open/close 10485 (time 1763341864.87 total 587.46 last 25.59) - open/close 10730 (time 1763341874.87 total 597.47 last 24.49) - open/close 10928 (time 1763341884.90 total 607.49 last 19.75) - open/close 11130 (time 1763341895.13 total 617.72 last 19.75) - open/close 11231 (time 1763341905.54 total 628.14 last 9.70) - open/close 11445 (time 1763341915.61 total 638.21 last 21.25) - open/close 11653 (time 1763341926.55 total 649.15 last 19.01) - open/close 11929 (time 1763341936.58 total 659.17 last 27.53) - open/close 12152 (time 1763341947.09 total 669.69 last 21.21) - open/close 12333 (time 1763341957.10 total 679.70 last 18.08) - open/close 12558 (time 1763341967.19 total 689.79 last 22.30) - open/close 12815 (time 1763341978.01 total 700.61 last 23.75) - open/close 13042 (time 1763341989.02 total 711.61 last 20.62) - open/close 13241 (time 1763341999.02 total 721.62 last 19.89) - open/close 13471 (time 1763342009.29 total 731.89 last 22.40) - open/close 13665 (time 1763342019.41 total 742.01 last 19.17) - open/close 13921 (time 1763342030.36 total 752.95 last 23.39) - open/close 14119 (time 1763342040.36 total 762.96 last 19.79) - open/close 14253 (time 1763342051.16 total 773.76 last 12.40) - open/close 14468 (time 1763342061.90 total 784.50 last 20.02) - open/close 14682 (time 1763342072.31 total 794.91 last 20.56) - open/close 14912 (time 1763342082.99 total 805.59 last 21.54) - open/close 15130 (time 1763342093.02 total 815.62 last 21.73) - open/close 15376 (time 1763342103.34 total 825.93 last 23.85) - open/close 15593 (time 1763342113.48 total 836.08 last 21.39) - open/close 15837 (time 1763342124.31 total 846.90 last 22.55) - open/close 16099 (time 1763342135.31 total 857.91 last 23.80) - open/close 16336 (time 1763342145.33 total 867.92 last 23.67) - open/close 16559 (time 1763342155.33 total 877.93 last 22.28) - open/close 16759 (time 1763342166.28 total 888.88 last 18.27) - open/close 16956 (time 1763342176.80 total 899.40 last 18.73) - open/close 17164 (time 1763342186.82 total 909.42 last 20.76) - open/close 17380 (time 1763342196.82 total 919.42 last 21.59) - open/close 17605 (time 1763342206.99 total 929.58 last 22.14) - open/close 18316 (time 1763342217.00 total 939.59 last 71.04) - open/close 19060 (time 1763342227.00 total 949.59 last 74.39) - open/close 19755 (time 1763342237.00 total 959.60 last 69.46) - open/close 20000 (time 1763342240.79 total 963.39 last 64.65) - open/close 20623 (time 1763342250.80 total 973.39 last 62.29) - open/close 21289 (time 1763342260.81 total 983.41 last 66.50) - open/close 21997 (time 1763342270.81 total 993.41 last 70.77) - open/close 22657 (time 1763342280.83 total 1003.42 last 65.90) - open/close 23295 (time 1763342290.84 total 1013.44 last 63.71) - open/close 23931 (time 1763342300.85 total 1023.45 last 63.55) - open/close 24607 (time 1763342310.86 total 1033.46 last 67.54) - open/close 25290 (time 1763342320.87 total 1043.46 last 68.26) - open/close 26003 (time 1763342330.87 total 1053.47 last 71.27) - open/close 26741 (time 1763342340.88 total 1063.47 last 73.76) - open/close 27348 (time 1763342350.90 total 1073.49 last 60.58) - open/close 28076 (time 1763342360.90 total 1083.49 last 72.79) - open/close 28735 (time 1763342370.91 total 1093.51 last 65.82) - open/close 29288 (time 1763342380.95 total 1103.54 last 55.10) - open/close 29837 (time 1763342390.95 total 1113.55 last 54.85) - open/close 30000 (time 1763342393.31 total 1115.91 last 69.20) - open/close 30685 (time 1763342403.32 total 1125.92 last 68.40) - open/close 31340 (time 1763342413.34 total 1135.93 last 65.41) - open/close 32089 (time 1763342423.35 total 1145.95 last 74.81) - open/close 32906 (time 1763342433.35 total 1155.95 last 81.70) - open/close 33552 (time 1763342443.35 total 1165.95 last 64.59) - open/close 34302 (time 1763342453.36 total 1175.95 last 74.97) - open/close 35062 (time 1763342463.36 total 1185.96 last 75.97) - open/close 35806 (time 1763342473.38 total 1195.97 last 74.28) - open/close 36589 (time 1763342483.39 total 1205.99 last 78.18) - open/close 37381 (time 1763342493.40 total 1215.99 last 79.15) - open/close 38128 (time 1763342503.41 total 1226.01 last 74.58) - open/close 38841 (time 1763342513.42 total 1236.02 last 71.22) - open/close 39394 (time 1763342523.44 total 1246.04 last 55.21) - open/close 40000 (time 1763342533.10 total 1255.70 last 62.72) - open/close 40702 (time 1763342543.11 total 1265.70 last 70.16) - open/close 41408 (time 1763342553.12 total 1275.71 last 70.54) - open/close 42029 (time 1763342563.13 total 1285.72 last 62.05) - open/close 42663 (time 1763342573.13 total 1295.72 last 63.38) - open/close 43360 (time 1763342583.13 total 1305.73 last 69.67) - open/close 44001 (time 1763342593.14 total 1315.73 last 64.08) - open/close 44741 (time 1763342603.15 total 1325.74 last 73.92) - open/close 45437 (time 1763342613.15 total 1335.74 last 69.59) - open/close 46129 (time 1763342623.15 total 1345.74 last 69.20) - open/close 46909 (time 1763342633.16 total 1355.75 last 77.95) - open/close 47720 (time 1763342643.16 total 1365.75 last 81.09) - open/close 48401 (time 1763342653.16 total 1375.75 last 68.09) - open/close 49148 (time 1763342663.17 total 1385.76 last 74.64) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 1392.54 seconds: 35.64 ops/second - unlinked 0 (time 1763342674 ; total 0 ; last 0) - unlinked 10000 (time 1763342808 ; total 134 ; last 134) - unlinked 20000 (time 1763342945 ; total 271 ; last 137) - unlinked 30000 (time 1763343080 ; total 406 ; last 135) - unlinked 40000 (time 1763343217 ; total 543 ; last 137) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 670 seconds: 74.077614 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg444-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg444-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg444-server stop mds service on oleg444-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg444-server stop mds service on oleg444-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg444-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg444-server' oleg444-server: oleg444-server.virtnet: executing unload_modules_local oleg444-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg444-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg444-client: oleg444-client: ssh exited with exit code 2 pdsh@oleg444-client: oleg444-server: ssh exited with exit code 2 pdsh@oleg444-client: oleg444-client: ssh exited with exit code 2 pdsh@oleg444-client: oleg444-server: ssh exited with exit code 2