== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:37:58 (1764736678) start mds service on oleg308-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg308-server' oleg308-server: oleg308-server.virtnet: executing load_modules_local oleg308-server: Loading modules from /home/green/git/lustre-release/lustre oleg308-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg308-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg308-server: detected 4 online CPUs by sysfs oleg308-server: MODOPTS_LIBCFS= oleg308-server: Force libcfs to create 2 CPU partitions oleg308-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg308-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg308-server: oleg308-server.virtnet: executing set_default_debug -1 all pdsh@oleg308-client: oleg308-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg308-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg308-server: oleg308-server.virtnet: executing set_default_debug -1 all pdsh@oleg308-client: oleg308-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg308-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg308-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg308-server: oleg308-server.virtnet: executing set_default_debug -1 all pdsh@oleg308-client: oleg308-server: ssh exited with exit code 1 Started lustre-OST0000 oleg308-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg308-client.virtnet: -o user_xattr,flock 192.168.203.108@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 192 (time 1764736745.35 total 10.04 last 19.12) - open/close 269 (time 1764736756.99 total 21.69 last 6.61) - open/close 463 (time 1764736767.04 total 31.74 last 19.31) - open/close 659 (time 1764736777.34 total 42.04 last 19.03) - open/close 868 (time 1764736787.72 total 52.41 last 20.15) - open/close 994 (time 1764736797.72 total 62.42 last 12.59) - open/close 1141 (time 1764736807.87 total 72.57 last 14.48) - open/close 1309 (time 1764736818.24 total 82.94 last 16.20) - open/close 1455 (time 1764736828.43 total 93.13 last 14.32) - open/close 1601 (time 1764736838.59 total 103.29 last 14.37) - open/close 1733 (time 1764736848.61 total 113.30 last 13.18) - open/close 1868 (time 1764736858.62 total 123.32 last 13.48) - open/close 1985 (time 1764736869.25 total 133.94 last 11.01) - open/close 2167 (time 1764736879.61 total 144.31 last 17.56) - open/close 2412 (time 1764736890.24 total 154.94 last 23.05) - open/close 2586 (time 1764736900.66 total 165.36 last 16.69) - open/close 2744 (time 1764736911.55 total 176.24 last 14.52) - open/close 2937 (time 1764736922.24 total 186.94 last 18.05) - open/close 3077 (time 1764736932.41 total 197.11 last 13.77) - open/close 3211 (time 1764736942.53 total 207.23 last 13.24) - open/close 3413 (time 1764736953.36 total 218.05 last 18.66) - open/close 3593 (time 1764736964.10 total 228.79 last 16.76) - open/close 3828 (time 1764736975.11 total 239.81 last 21.33) - open/close 3964 (time 1764736985.13 total 249.82 last 13.58) - open/close 4170 (time 1764736996.09 total 260.79 last 18.79) - open/close 4316 (time 1764737006.10 total 270.80 last 14.59) - open/close 4426 (time 1764737016.83 total 281.52 last 10.25) - open/close 4528 (time 1764737027.40 total 292.10 last 9.65) - open/close 4803 (time 1764737038.33 total 303.03 last 25.16) - open/close 4928 (time 1764737049.08 total 313.78 last 11.63) - open/close 5089 (time 1764737059.16 total 323.85 last 15.98) - open/close 5254 (time 1764737069.50 total 334.20 last 15.96) - open/close 5377 (time 1764737080.45 total 345.14 last 11.24) - open/close 5493 (time 1764737091.52 total 356.22 last 10.47) - open/close 5801 (time 1764737101.70 total 366.39 last 30.28) - open/close 6057 (time 1764737111.70 total 376.39 last 25.60) - open/close 6333 (time 1764737122.11 total 386.81 last 26.50) - open/close 6610 (time 1764737133.76 total 398.45 last 23.79) - open/close 6923 (time 1764737144.44 total 409.14 last 29.29) - open/close 7241 (time 1764737155.20 total 419.90 last 29.56) - open/close 7565 (time 1764737165.20 total 429.90 last 32.40) - open/close 7839 (time 1764737175.74 total 440.43 last 26.01) - open/close 8077 (time 1764737186.13 total 450.83 last 22.90) - open/close 8252 (time 1764737196.23 total 460.92 last 17.33) - open/close 8574 (time 1764737206.41 total 471.10 last 31.62) - open/close 8892 (time 1764737217.28 total 481.97 last 29.26) - open/close 9174 (time 1764737227.77 total 492.47 last 26.87) - open/close 9469 (time 1764737237.94 total 502.64 last 29.00) - open/close 9702 (time 1764737248.51 total 513.20 last 22.05) - open/close 9963 (time 1764737259.51 total 524.21 last 23.72) - open/close 10246 (time 1764737269.95 total 534.64 last 27.12) - open/close 10512 (time 1764737279.95 total 544.64 last 26.60) - open/close 11839 (time 1764737289.96 total 554.65 last 132.61) - open/close 13343 (time 1764737299.96 total 564.66 last 150.30) - open/close 14898 (time 1764737309.97 total 574.67 last 155.33) - open/close 16312 (time 1764737319.98 total 584.67 last 141.38) - open/close 17675 (time 1764737329.98 total 594.67 last 136.26) - open/close 19062 (time 1764737339.98 total 604.67 last 138.68) - open/close 20000 (time 1764737346.65 total 611.35 last 140.55) - open/close 21629 (time 1764737356.66 total 621.35 last 162.85) - open/close 23425 (time 1764737366.66 total 631.35 last 179.57) - open/close 25083 (time 1764737376.66 total 641.36 last 165.74) - open/close 26712 (time 1764737386.67 total 651.36 last 162.83) - open/close 28169 (time 1764737396.67 total 661.37 last 145.64) - open/close 29857 (time 1764737406.68 total 671.37 last 168.66) - open/close 31323 (time 1764737416.68 total 681.37 last 146.60) - open/close 33167 (time 1764737426.68 total 691.38 last 184.31) - open/close 34635 (time 1764737436.69 total 701.38 last 146.74) - open/close 36362 (time 1764737446.69 total 711.39 last 172.64) - open/close 37913 (time 1764737456.70 total 721.40 last 154.92) - open/close 39564 (time 1764737466.71 total 731.40 last 165.01) - open/close 40000 (time 1764737469.11 total 733.81 last 181.48) - open/close 42003 (time 1764737479.12 total 743.81 last 200.14) - open/close 43868 (time 1764737489.12 total 753.82 last 186.46) - open/close 46179 (time 1764737499.12 total 763.82 last 231.07) - open/close 48477 (time 1764737509.12 total 773.82 last 229.78) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 778.73 seconds: 63.73 ops/second - unlinked 0 (time 1764737515 ; total 0 ; last 0) - unlinked 10000 (time 1764737541 ; total 26 ; last 26) - unlinked 20000 (time 1764737565 ; total 50 ; last 24) - unlinked 30000 (time 1764737589 ; total 74 ; last 24) - unlinked 40000 (time 1764737611 ; total 96 ; last 22) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 117 seconds: 424.205139 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg308-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg308-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg308-server stop mds service on oleg308-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg308-server stop mds service on oleg308-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg308-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg308-server' oleg308-server: oleg308-server.virtnet: executing unload_modules_local oleg308-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg308-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg308-client: oleg308-client: ssh exited with exit code 2 pdsh@oleg308-client: oleg308-server: ssh exited with exit code 2 pdsh@oleg308-client: oleg308-client: ssh exited with exit code 2 pdsh@oleg308-client: oleg308-server: ssh exited with exit code 2