== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 02:29:25 (1764746965) start mds service on oleg118-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg118-server' oleg118-server: oleg118-server.virtnet: executing load_modules_local oleg118-server: Loading modules from /home/green/git/lustre-release/lustre oleg118-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg118-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg118-server: detected 4 online CPUs by sysfs oleg118-server: MODOPTS_LIBCFS= oleg118-server: Force libcfs to create 2 CPU partitions oleg118-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg118-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg118-server: oleg118-server.virtnet: executing set_default_debug -1 all pdsh@oleg118-client: oleg118-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg118-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg118-server: oleg118-server.virtnet: executing set_default_debug -1 all pdsh@oleg118-client: oleg118-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg118-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg118-server: oleg118-server.virtnet: executing set_default_debug -1 all pdsh@oleg118-client: oleg118-server: ssh exited with exit code 1 Started lustre-OST0000 oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg118-client.virtnet: -o user_xattr,flock 192.168.201.118@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 321 (time 1764747028.33 total 10.41 last 30.85) - open/close 504 (time 1764747038.83 total 20.91 last 17.43) - open/close 731 (time 1764747049.45 total 31.53 last 21.37) - open/close 940 (time 1764747059.94 total 42.02 last 19.92) - open/close 1174 (time 1764747070.63 total 52.71 last 21.89) - open/close 1373 (time 1764747081.00 total 63.08 last 19.19) - open/close 1628 (time 1764747091.94 total 74.02 last 23.31) - open/close 1942 (time 1764747102.12 total 84.20 last 30.86) - open/close 2189 (time 1764747112.87 total 94.95 last 22.96) - open/close 2453 (time 1764747122.88 total 104.95 last 26.39) - open/close 2607 (time 1764747132.91 total 114.99 last 15.35) - open/close 2837 (time 1764747143.66 total 125.74 last 21.39) - open/close 2980 (time 1764747153.77 total 135.84 last 14.15) - open/close 3510 (time 1764747164.77 total 146.85 last 48.15) - open/close 3690 (time 1764747174.89 total 156.97 last 17.79) - open/close 3935 (time 1764747185.52 total 167.60 last 23.05) - open/close 4210 (time 1764747195.62 total 177.70 last 27.22) - open/close 4526 (time 1764747206.44 total 188.52 last 29.21) - open/close 4803 (time 1764747217.19 total 199.27 last 25.76) - open/close 4899 (time 1764747227.95 total 210.02 last 8.93) - open/close 5144 (time 1764747238.38 total 220.46 last 23.48) - open/close 5363 (time 1764747248.68 total 230.76 last 21.26) - open/close 5570 (time 1764747258.92 total 241.00 last 20.21) - open/close 5915 (time 1764747268.93 total 251.01 last 34.46) - open/close 6134 (time 1764747279.27 total 261.35 last 21.17) - open/close 6373 (time 1764747290.09 total 272.17 last 22.10) - open/close 6760 (time 1764747301.10 total 283.18 last 35.14) - open/close 7058 (time 1764747311.53 total 293.61 last 28.58) - open/close 7299 (time 1764747322.54 total 304.62 last 21.89) - open/close 7560 (time 1764747333.03 total 315.11 last 24.87) - open/close 7699 (time 1764747343.04 total 325.11 last 13.90) - open/close 7853 (time 1764747353.26 total 335.34 last 15.06) - open/close 8135 (time 1764747364.20 total 346.28 last 25.76) - open/close 8399 (time 1764747375.14 total 357.22 last 24.13) - open/close 8628 (time 1764747385.77 total 367.85 last 21.56) - open/close 8851 (time 1764747396.27 total 378.35 last 21.23) - open/close 9132 (time 1764747406.27 total 388.35 last 28.09) - open/close 10000 (time 1764747411.54 total 393.61 last 164.98) - open/close 11811 (time 1764747421.54 total 403.62 last 181.01) - open/close 13827 (time 1764747431.55 total 413.62 last 201.51) - open/close 15950 (time 1764747441.55 total 423.62 last 212.28) - open/close 18621 (time 1764747451.55 total 433.62 last 267.09) - open/close 20000 (time 1764747456.40 total 438.47 last 284.38) - open/close 22544 (time 1764747466.40 total 448.48 last 254.34) - open/close 25406 (time 1764747476.40 total 458.48 last 286.03) - open/close 27096 (time 1764747486.41 total 468.48 last 168.97) - open/close 29017 (time 1764747496.41 total 478.49 last 192.01) - open/close 30000 (time 1764747499.94 total 482.01 last 278.83) - open/close 32624 (time 1764747509.94 total 492.02 last 262.30) - open/close 35930 (time 1764747519.94 total 502.02 last 330.58) - open/close 39050 (time 1764747529.94 total 512.02 last 311.91) - open/close 40000 (time 1764747533.31 total 515.39 last 282.26) - open/close 43374 (time 1764747543.31 total 525.39 last 337.32) - open/close 46576 (time 1764747553.31 total 535.39 last 320.11) - open/close 49458 (time 1764747563.32 total 545.39 last 288.15) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 546.31 seconds: 90.85 ops/second - unlinked 0 (time 1764747566 ; total 0 ; last 0) - unlinked 10000 (time 1764747600 ; total 34 ; last 34) - unlinked 20000 (time 1764747622 ; total 56 ; last 22) - unlinked 30000 (time 1764747648 ; total 82 ; last 26) - unlinked 40000 (time 1764747670 ; total 104 ; last 22) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 126 seconds: 393.904755 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg118-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg118-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg118-server stop mds service on oleg118-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg118-server stop mds service on oleg118-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg118-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg118-server' oleg118-server: oleg118-server.virtnet: executing unload_modules_local oleg118-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg118-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg118-client: oleg118-client: ssh exited with exit code 2 pdsh@oleg118-client: oleg118-server: ssh exited with exit code 2 pdsh@oleg118-client: oleg118-server: ssh exited with exit code 2 pdsh@oleg118-client: oleg118-client: ssh exited with exit code 2