== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 11:33:54 (1754408034) start mds service on oleg218-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg218-server' oleg218-server: oleg218-server.virtnet: executing load_modules_local oleg218-server: Loading modules from /home/green/git/lustre-release/lustre oleg218-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy oleg218-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy oleg218-server: detected 4 online CPUs by sysfs oleg218-server: MODOPTS_LIBCFS= oleg218-server: Force libcfs to create 2 CPU partitions oleg218-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg218-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg218-server: oleg218-server.virtnet: executing set_default_debug -1 all pdsh@oleg218-client: oleg218-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg218-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg218-server: oleg218-server.virtnet: executing set_default_debug -1 all pdsh@oleg218-client: oleg218-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg218-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg218-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg218-server: oleg218-server.virtnet: executing set_default_debug -1 all pdsh@oleg218-client: oleg218-server: ssh exited with exit code 1 Started lustre-OST0000 oleg218-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg218-client.virtnet: -o user_xattr,flock 192.168.202.118@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 155 (time 1754408093.51 total 10.01 last 15.49) - open/close 390 (time 1754408103.98 total 20.48 last 22.44) - open/close 585 (time 1754408114.39 total 30.90 last 18.72) - open/close 878 (time 1754408124.39 total 40.90 last 29.30) - open/close 1662 (time 1754408134.81 total 51.31 last 75.30) - open/close 1948 (time 1754408145.24 total 61.74 last 27.41) - open/close 2212 (time 1754408155.74 total 72.24 last 25.15) - open/close 2431 (time 1754408168.22 total 84.72 last 17.55) - open/close 2750 (time 1754408179.23 total 95.73 last 28.99) - open/close 2960 (time 1754408190.23 total 106.73 last 19.08) - open/close 3173 (time 1754408200.23 total 116.74 last 21.30) - open/close 3585 (time 1754408210.23 total 126.74 last 41.19) - open/close 3836 (time 1754408221.02 total 137.52 last 23.27) - open/close 4064 (time 1754408231.03 total 147.54 last 22.77) - open/close 4349 (time 1754408241.49 total 157.99 last 27.25) - open/close 4553 (time 1754408251.56 total 168.06 last 20.27) - open/close 4876 (time 1754408261.59 total 178.09 last 32.19) - open/close 5140 (time 1754408271.59 total 188.10 last 26.39) - open/close 5436 (time 1754408282.45 total 198.96 last 27.26) - open/close 5697 (time 1754408292.46 total 208.96 last 26.08) - open/close 5967 (time 1754408303.26 total 219.76 last 25.00) - open/close 6218 (time 1754408313.26 total 229.76 last 25.10) - open/close 6493 (time 1754408323.55 total 240.05 last 26.73) - open/close 6739 (time 1754408333.78 total 250.29 last 24.03) - open/close 7030 (time 1754408344.54 total 261.04 last 27.06) - open/close 7282 (time 1754408355.03 total 271.53 last 24.02) - open/close 7576 (time 1754408365.52 total 282.03 last 28.01) - open/close 7751 (time 1754408375.77 total 292.27 last 17.09) - open/close 7878 (time 1754408386.64 total 303.14 last 11.68) - open/close 8145 (time 1754408397.27 total 313.77 last 25.12) - open/close 8492 (time 1754408408.08 total 324.59 last 32.09) - open/close 8768 (time 1754408418.71 total 335.22 last 25.96) - open/close 9017 (time 1754408429.53 total 346.03 last 23.02) - open/close 9272 (time 1754408440.34 total 356.85 last 23.58) - open/close 9536 (time 1754408450.35 total 366.85 last 26.39) - open/close 9743 (time 1754408460.82 total 377.32 last 19.76) - open/close 9991 (time 1754408471.32 total 387.82 last 23.63) - open/close 10271 (time 1754408481.81 total 398.32 last 26.68) - open/close 10491 (time 1754408492.24 total 408.75 last 21.09) - open/close 10688 (time 1754408502.68 total 419.18 last 18.89) - open/close 11067 (time 1754408513.36 total 429.87 last 35.46) - open/close 12588 (time 1754408523.37 total 439.87 last 152.05) - open/close 13957 (time 1754408533.37 total 449.87 last 136.87) - open/close 15112 (time 1754408543.37 total 459.87 last 115.46) - open/close 16550 (time 1754408553.37 total 469.88 last 143.75) - open/close 18715 (time 1754408563.38 total 479.88 last 216.38) - open/close 20000 (time 1754408568.92 total 485.42 last 232.07) - open/close 21836 (time 1754408578.92 total 495.42 last 183.57) - open/close 23650 (time 1754408588.92 total 505.42 last 181.35) - open/close 25235 (time 1754408598.93 total 515.43 last 158.40) - open/close 26855 (time 1754408608.93 total 525.43 last 161.95) - open/close 28670 (time 1754408618.93 total 535.43 last 181.48) - open/close 30000 (time 1754408624.58 total 541.08 last 235.66) - open/close 32319 (time 1754408634.58 total 551.08 last 231.85) - open/close 34149 (time 1754408644.58 total 561.09 last 182.91) - open/close 36652 (time 1754408654.58 total 571.09 last 250.26) - open/close 38821 (time 1754408664.59 total 581.09 last 216.89) - open/close 40000 (time 1754408669.71 total 586.21 last 230.29) - open/close 42032 (time 1754408679.71 total 596.21 last 203.08) - open/close 43082 (time 1754408689.72 total 606.22 last 104.93) - open/close 44447 (time 1754408699.72 total 616.23 last 136.41) - open/close 45744 (time 1754408709.73 total 626.23 last 129.64) - open/close 47404 (time 1754408719.73 total 636.24 last 165.93) - open/close 48904 (time 1754408729.73 total 646.24 last 149.97) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 650.50 seconds: 76.30 ops/second - unlinked 0 (time 1754408735 ; total 0 ; last 0) - unlinked 10000 (time 1754408770 ; total 35 ; last 35) - unlinked 20000 (time 1754408800 ; total 65 ; last 30) - unlinked 30000 (time 1754408829 ; total 94 ; last 29) - unlinked 40000 (time 1754408858 ; total 123 ; last 29) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 148 seconds: 335.351349 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg218-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg218-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg218-server stop mds service on oleg218-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg218-server stop mds service on oleg218-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg218-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg218-server' oleg218-server: oleg218-server.virtnet: executing unload_modules_local oleg218-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg218-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg218-client: oleg218-client: ssh exited with exit code 2 pdsh@oleg218-client: oleg218-server: ssh exited with exit code 2 pdsh@oleg218-client: oleg218-client: ssh exited with exit code 2 pdsh@oleg218-client: oleg218-server: ssh exited with exit code 2