== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 19:35:49 (1755041749) start mds service on oleg208-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg208-server' oleg208-server: oleg208-server.virtnet: executing load_modules_local oleg208-server: Loading modules from /home/green/git/lustre-release/lustre oleg208-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy oleg208-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy oleg208-server: detected 4 online CPUs by sysfs oleg208-server: MODOPTS_LIBCFS= oleg208-server: Force libcfs to create 2 CPU partitions oleg208-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg208-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg208-server: oleg208-server.virtnet: executing set_default_debug -1 all pdsh@oleg208-client: oleg208-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg208-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg208-server: oleg208-server.virtnet: executing set_default_debug -1 all pdsh@oleg208-client: oleg208-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg208-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg208-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg208-server: oleg208-server.virtnet: executing set_default_debug -1 all pdsh@oleg208-client: oleg208-server: ssh exited with exit code 1 Started lustre-OST0000 oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg208-client.virtnet: -o user_xattr,flock 192.168.202.108@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 151 (time 1755041846.47 total 10.02 last 15.07) - open/close 398 (time 1755041857.53 total 21.07 last 22.35) - open/close 619 (time 1755041867.57 total 31.11 last 22.01) - open/close 842 (time 1755041877.74 total 41.28 last 21.92) - open/close 1073 (time 1755041888.17 total 51.72 last 22.14) - open/close 1287 (time 1755041898.18 total 61.73 last 21.37) - open/close 1551 (time 1755041909.09 total 72.64 last 24.19) - open/close 1768 (time 1755041919.10 total 82.65 last 21.68) - open/close 1959 (time 1755041930.16 total 93.71 last 17.27) - open/close 2164 (time 1755041940.17 total 103.72 last 20.48) - open/close 2416 (time 1755041950.18 total 113.73 last 25.18) - open/close 2652 (time 1755041960.20 total 123.75 last 23.55) - open/close 2788 (time 1755041971.28 total 134.83 last 12.27) - open/close 3047 (time 1755041984.36 total 147.91 last 19.80) - open/close 3191 (time 1755041994.88 total 158.42 last 13.69) - open/close 3293 (time 1755042004.93 total 168.47 last 10.15) - open/close 3410 (time 1755042015.24 total 178.78 last 11.35) - open/close 3482 (time 1755042025.73 total 189.28 last 6.86) - open/close 3628 (time 1755042036.10 total 199.64 last 14.09) - open/close 3748 (time 1755042046.83 total 210.38 last 11.18) - open/close 3837 (time 1755042057.57 total 221.12 last 8.29) - open/close 3999 (time 1755042067.59 total 231.13 last 16.18) - open/close 4145 (time 1755042078.56 total 242.11 last 13.31) - open/close 4283 (time 1755042089.37 total 252.91 last 12.77) - open/close 4510 (time 1755042099.38 total 262.92 last 22.68) - open/close 4726 (time 1755042110.46 total 274.01 last 19.49) - open/close 4932 (time 1755042121.20 total 284.75 last 19.18) - open/close 5081 (time 1755042131.28 total 294.82 last 14.79) - open/close 5185 (time 1755042141.29 total 304.84 last 10.39) - open/close 5431 (time 1755042151.30 total 314.84 last 24.58) - open/close 5634 (time 1755042161.33 total 324.87 last 20.24) - open/close 5711 (time 1755042171.66 total 335.20 last 7.45) - open/close 5815 (time 1755042182.35 total 345.90 last 9.73) - open/close 5898 (time 1755042193.32 total 356.87 last 7.57) - open/close 6087 (time 1755042204.15 total 367.69 last 17.46) - open/close 6253 (time 1755042214.20 total 377.74 last 16.52) - open/close 6468 (time 1755042224.94 total 388.48 last 20.02) - open/close 6691 (time 1755042234.94 total 398.49 last 22.30) - open/close 6944 (time 1755042245.24 total 408.79 last 24.55) - open/close 7111 (time 1755042255.98 total 419.52 last 15.56) - open/close 7229 (time 1755042266.21 total 429.76 last 11.53) - open/close 7400 (time 1755042277.10 total 440.64 last 15.71) - open/close 7550 (time 1755042287.20 total 450.75 last 14.85) - open/close 7645 (time 1755042297.83 total 461.37 last 8.94) - open/close 7793 (time 1755042308.40 total 471.95 last 14.00) - open/close 7925 (time 1755042318.64 total 482.18 last 12.90) - open/close 8074 (time 1755042329.07 total 492.62 last 14.28) - open/close 8257 (time 1755042339.09 total 502.64 last 18.27) - open/close 8408 (time 1755042349.74 total 513.28 last 14.19) - open/close 8593 (time 1755042360.67 total 524.22 last 16.91) - open/close 8785 (time 1755042370.68 total 534.23 last 19.19) - open/close 8894 (time 1755042381.67 total 545.21 last 9.92) - open/close 9062 (time 1755042392.16 total 555.71 last 16.01) - open/close 9297 (time 1755042402.18 total 565.73 last 23.45) - open/close 9422 (time 1755042412.46 total 576.00 last 12.16) - open/close 9587 (time 1755042423.14 total 586.69 last 15.45) - open/close 9750 (time 1755042433.15 total 596.69 last 16.28) - open/close 10000 (time 1755042443.02 total 606.56 last 25.33) - open/close 10198 (time 1755042453.87 total 617.41 last 18.25) - open/close 10392 (time 1755042464.22 total 627.77 last 18.74) - open/close 10638 (time 1755042474.66 total 638.21 last 23.56) - open/close 10916 (time 1755042485.35 total 648.89 last 26.02) - open/close 11125 (time 1755042495.66 total 659.21 last 20.26) - open/close 11305 (time 1755042506.42 total 669.97 last 16.73) - open/close 11526 (time 1755042516.42 total 679.97 last 22.10) - open/close 11708 (time 1755042526.94 total 690.49 last 17.30) - open/close 11947 (time 1755042537.95 total 701.49 last 21.71) - open/close 12209 (time 1755042547.95 total 711.50 last 26.19) - open/close 12453 (time 1755042557.95 total 721.50 last 24.40) - open/close 12728 (time 1755042567.95 total 731.50 last 27.50) - open/close 13546 (time 1755042577.96 total 741.51 last 81.73) - open/close 15263 (time 1755042587.97 total 751.51 last 171.57) - open/close 16841 (time 1755042597.97 total 761.52 last 157.76) - open/close 18335 (time 1755042607.97 total 771.52 last 149.35) - open/close 19732 (time 1755042617.98 total 781.52 last 139.65) - open/close 21317 (time 1755042627.98 total 791.53 last 158.47) - open/close 22615 (time 1755042637.98 total 801.53 last 129.76) - open/close 23762 (time 1755042647.98 total 811.53 last 114.69) - open/close 24932 (time 1755042657.99 total 821.54 last 116.88) - open/close 25978 (time 1755042668.00 total 831.54 last 104.57) - open/close 26956 (time 1755042678.00 total 841.55 last 97.74) - open/close 28270 (time 1755042688.01 total 851.55 last 131.32) - open/close 29373 (time 1755042698.01 total 861.56 last 110.25) - open/close 30000 (time 1755042703.14 total 866.69 last 122.27) - open/close 31178 (time 1755042713.14 total 876.69 last 117.78) - open/close 32832 (time 1755042723.15 total 886.69 last 165.30) - open/close 34833 (time 1755042733.15 total 896.70 last 200.05) - open/close 36914 (time 1755042743.15 total 906.70 last 208.01) - open/close 38562 (time 1755042753.16 total 916.71 last 164.67) - open/close 39706 (time 1755042763.16 total 926.71 last 114.39) - open/close 40000 (time 1755042765.21 total 928.75 last 143.92) - open/close 41246 (time 1755042775.21 total 938.76 last 124.50) - open/close 42280 (time 1755042785.22 total 948.77 last 103.32) - open/close 43248 (time 1755042795.23 total 958.78 last 96.69) - open/close 44307 (time 1755042805.24 total 968.78 last 105.84) - open/close 45168 (time 1755042815.24 total 978.79 last 86.07) - open/close 46008 (time 1755042825.25 total 988.79 last 83.96) - open/close 46806 (time 1755042835.25 total 998.79 last 79.79) - open/close 47597 (time 1755042845.26 total 1008.80 last 79.02) - open/close 48416 (time 1755042855.27 total 1018.81 last 81.83) - open/close 49390 (time 1755042865.27 total 1028.82 last 97.33) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 1031.41 seconds: 48.12 ops/second - unlinked 0 (time 1755042870 ; total 0 ; last 0) - unlinked 10000 (time 1755042954 ; total 84 ; last 84) - unlinked 20000 (time 1755042975 ; total 105 ; last 21) - unlinked 30000 (time 1755043000 ; total 130 ; last 25) - unlinked 40000 (time 1755043039 ; total 169 ; last 39) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 204 seconds: 243.294113 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg208-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg208-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg208-server stop mds service on oleg208-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg208-server stop mds service on oleg208-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg208-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg208-server' oleg208-server: oleg208-server.virtnet: executing unload_modules_local oleg208-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg208-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg208-client: oleg208-server: ssh exited with exit code 2 pdsh@oleg208-client: oleg208-client: ssh exited with exit code 2 pdsh@oleg208-client: oleg208-client: ssh exited with exit code 2 pdsh@oleg208-client: oleg208-server: ssh exited with exit code 2