== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 07:18:24 (1756811904) start mds service on oleg415-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg415-server' oleg415-server: oleg415-server.virtnet: executing load_modules_local oleg415-server: Loading modules from /home/green/git/lustre-release/lustre oleg415-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg415-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg415-server: detected 4 online CPUs by sysfs oleg415-server: MODOPTS_LIBCFS= oleg415-server: Force libcfs to create 2 CPU partitions oleg415-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg415-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg415-server: oleg415-server.virtnet: executing set_default_debug -1 all pdsh@oleg415-client: oleg415-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg415-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg415-server: oleg415-server.virtnet: executing set_default_debug -1 all pdsh@oleg415-client: oleg415-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg415-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg415-server: oleg415-server.virtnet: executing set_default_debug -1 all pdsh@oleg415-client: oleg415-server: ssh exited with exit code 1 Started lustre-OST0000 oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg415-client.virtnet: -o user_xattr,flock 192.168.204.115@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 747 (time 1756811956.48 total 10.34 last 72.23) - open/close 1540 (time 1756811966.85 total 20.71 last 76.47) - open/close 2179 (time 1756811977.02 total 30.89 last 62.80) - open/close 2961 (time 1756811987.20 total 41.06 last 76.87) - open/close 3683 (time 1756811997.50 total 51.37 last 70.07) - open/close 4350 (time 1756812007.93 total 61.80 last 63.93) - open/close 5028 (time 1756812018.68 total 72.55 last 63.06) - open/close 5795 (time 1756812029.57 total 83.43 last 70.49) - open/close 6614 (time 1756812039.57 total 93.43 last 81.90) - open/close 7358 (time 1756812050.49 total 104.36 last 68.08) - open/close 8314 (time 1756812060.99 total 114.86 last 91.09) - open/close 9280 (time 1756812071.68 total 125.54 last 90.40) - open/close 10000 (time 1756812079.50 total 133.37 last 92.01) - open/close 10833 (time 1756812089.54 total 143.40 last 83.01) - open/close 11739 (time 1756812099.84 total 153.71 last 87.92) - open/close 12657 (time 1756812110.21 total 164.07 last 88.57) - open/close 13590 (time 1756812120.51 total 174.37 last 90.55) - open/close 14475 (time 1756812130.94 total 184.81 last 84.83) - open/close 15258 (time 1756812140.99 total 194.86 last 77.92) - open/close 16083 (time 1756812151.11 total 204.97 last 81.55) - open/close 16973 (time 1756812161.41 total 215.27 last 86.41) - open/close 17763 (time 1756812172.42 total 226.28 last 71.75) - open/close 18532 (time 1756812183.29 total 237.16 last 70.71) - open/close 19180 (time 1756812193.92 total 247.78 last 60.98) - open/close 19807 (time 1756812204.54 total 258.41 last 59.03) - open/close 20571 (time 1756812214.54 total 268.41 last 76.39) - open/close 21336 (time 1756812224.54 total 278.41 last 76.50) - open/close 22059 (time 1756812234.68 total 288.55 last 71.29) - open/close 22635 (time 1756812245.12 total 298.98 last 55.20) - open/close 23256 (time 1756812255.74 total 309.61 last 58.47) - open/close 26231 (time 1756812265.74 total 319.61 last 297.45) - open/close 30000 (time 1756812274.08 total 327.95 last 452.02) - open/close 34651 (time 1756812284.08 total 337.95 last 464.95) - open/close 38989 (time 1756812294.10 total 347.97 last 433.05) - open/close 40000 (time 1756812296.56 total 350.42 last 411.27) - open/close 44928 (time 1756812306.56 total 360.42 last 492.78) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 370.15 seconds: 134.09 ops/second - unlinked 0 (time 1756812317 ; total 0 ; last 0) - unlinked 10000 (time 1756812329 ; total 12 ; last 12) - unlinked 20000 (time 1756812341 ; total 24 ; last 12) - unlinked 30000 (time 1756812353 ; total 36 ; last 12) - unlinked 40000 (time 1756812365 ; total 48 ; last 12) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 60 seconds: 827.200012 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg415-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg415-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg415-server stop mds service on oleg415-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg415-server stop mds service on oleg415-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg415-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg415-server' oleg415-server: oleg415-server.virtnet: executing unload_modules_local oleg415-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg415-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg415-client: oleg415-client: ssh exited with exit code 2 pdsh@oleg415-client: oleg415-server: ssh exited with exit code 2 pdsh@oleg415-client: oleg415-server: ssh exited with exit code 2 pdsh@oleg415-client: oleg415-client: ssh exited with exit code 2