== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 17:17:50 (1755638270) start mds service on oleg457-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg457-server' oleg457-server: oleg457-server.virtnet: executing load_modules_local oleg457-server: Loading modules from /home/green/git/lustre-release/lustre oleg457-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy oleg457-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy oleg457-server: detected 4 online CPUs by sysfs oleg457-server: MODOPTS_LIBCFS= oleg457-server: Force libcfs to create 2 CPU partitions oleg457-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg457-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug -1 all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg457-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg457-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg457-server: oleg457-server.virtnet: executing set_default_debug -1 all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-OST0000 oleg457-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg457-client.virtnet: -o user_xattr,flock 192.168.204.157@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 742 (time 1755638312.03 total 10.08 last 73.60) - open/close 1480 (time 1755638322.03 total 20.08 last 73.79) - open/close 2215 (time 1755638333.03 total 31.07 last 66.86) - open/close 3014 (time 1755638343.14 total 41.19 last 79.03) - open/close 3816 (time 1755638353.25 total 51.30 last 79.31) - open/close 4523 (time 1755638363.25 total 61.30 last 70.69) - open/close 5229 (time 1755638373.25 total 71.30 last 70.58) - open/close 5942 (time 1755638383.27 total 81.32 last 71.20) - open/close 6781 (time 1755638393.63 total 91.68 last 80.93) - open/close 7486 (time 1755638403.64 total 101.68 last 70.49) - open/close 8242 (time 1755638414.63 total 112.68 last 68.77) - open/close 8990 (time 1755638424.63 total 122.68 last 74.79) - open/close 9748 (time 1755638434.66 total 132.71 last 75.58) - open/close 10000 (time 1755638438.33 total 136.38 last 68.67) - open/close 10792 (time 1755638448.48 total 146.53 last 77.97) - open/close 11549 (time 1755638458.60 total 156.64 last 74.86) - open/close 12269 (time 1755638468.60 total 166.64 last 72.00) - open/close 12965 (time 1755638478.63 total 176.67 last 69.40) - open/close 13291 (time 1755638488.63 total 186.68 last 32.59) - open/close 14077 (time 1755638499.62 total 197.67 last 71.49) - open/close 14805 (time 1755638509.67 total 207.72 last 72.46) - open/close 15469 (time 1755638520.61 total 218.66 last 60.69) - open/close 16224 (time 1755638530.66 total 228.71 last 75.10) - open/close 16824 (time 1755638540.66 total 238.71 last 59.99) - open/close 17537 (time 1755638550.67 total 248.71 last 71.29) - open/close 18218 (time 1755638560.74 total 258.79 last 67.56) - open/close 18925 (time 1755638571.62 total 269.67 last 65.02) - open/close 19674 (time 1755638581.73 total 279.78 last 74.08) - open/close 20000 (time 1755638590.83 total 288.87 last 35.84) - open/close 20515 (time 1755638603.49 total 301.54 last 40.66) - open/close 21185 (time 1755638613.73 total 311.78 last 65.43) - open/close 22214 (time 1755638623.73 total 321.78 last 102.88) - open/close 25916 (time 1755638633.73 total 331.78 last 370.19) - open/close 28981 (time 1755638643.74 total 341.78 last 306.39) - open/close 30000 (time 1755638646.77 total 344.82 last 336.15) - open/close 33555 (time 1755638656.77 total 354.82 last 355.45) - open/close 37945 (time 1755638666.77 total 364.82 last 438.98) - open/close 40000 (time 1755638671.82 total 369.87 last 406.75) - open/close 43874 (time 1755638681.82 total 379.87 last 387.38) - open/close 47918 (time 1755638691.83 total 389.87 last 404.25) total: 50000 open/close in 397.30 seconds: 125.85 ops/second - unlinked 0 (time 1755638700 ; total 0 ; last 0) - unlinked 10000 (time 1755638716 ; total 16 ; last 16) - unlinked 20000 (time 1755638733 ; total 33 ; last 17) - unlinked 30000 (time 1755638749 ; total 49 ; last 16) - unlinked 40000 (time 1755638765 ; total 65 ; last 16) total: 50000 unlinks in 84 seconds: 595.238098 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg457-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg457-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg457-server stop mds service on oleg457-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg457-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg457-server' oleg457-server: oleg457-server.virtnet: executing unload_modules_local oleg457-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg457-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg457-client: oleg457-server: ssh exited with exit code 2 pdsh@oleg457-client: oleg457-client: ssh exited with exit code 2 pdsh@oleg457-client: oleg457-client: ssh exited with exit code 2 pdsh@oleg457-client: oleg457-server: ssh exited with exit code 2