== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 20:40:26 (1770687626) start mds service on oleg255-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg255-server' oleg255-server: oleg255-server.virtnet: executing load_modules_local oleg255-server: Loading modules from /home/green/git/lustre-release/lustre oleg255-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg255-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg255-server: detected 4 online CPUs by sysfs oleg255-server: MODOPTS_LIBCFS= oleg255-server: Force libcfs to create 2 CPU partitions oleg255-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg255-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg255-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug -1 all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg255-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg255-server: oleg255-server.virtnet: executing set_default_debug -1 all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Started lustre-OST0000 oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg255-client.virtnet: -o user_xattr,flock 192.168.202.155@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 216 (time 1770687678.34 total 10.60 last 20.38) - open/close 476 (time 1770687688.64 total 20.90 last 25.24) - open/close 733 (time 1770687698.88 total 31.14 last 25.09) - open/close 1027 (time 1770687709.51 total 41.77 last 27.67) - open/close 1297 (time 1770687720.13 total 52.39 last 25.42) - open/close 1605 (time 1770687730.77 total 63.03 last 28.94) - open/close 1934 (time 1770687740.87 total 73.14 last 32.57) - open/close 2129 (time 1770687750.98 total 83.24 last 19.30) - open/close 2415 (time 1770687761.22 total 93.48 last 27.92) - open/close 2687 (time 1770687771.41 total 103.67 last 26.70) - open/close 3014 (time 1770687781.44 total 113.70 last 32.60) - open/close 3401 (time 1770687791.81 total 124.08 last 37.30) - open/close 3692 (time 1770687802.18 total 134.44 last 28.08) - open/close 4000 (time 1770687812.80 total 145.06 last 28.98) - open/close 4291 (time 1770687824.58 total 156.84 last 24.70) - open/close 4608 (time 1770687835.52 total 167.78 last 28.99) - open/close 4863 (time 1770687846.40 total 178.66 last 23.44) - open/close 5136 (time 1770687856.40 total 188.67 last 27.28) - open/close 5514 (time 1770687866.81 total 199.08 last 36.31) - open/close 5789 (time 1770687876.99 total 209.26 last 27.01) - open/close 6039 (time 1770687887.11 total 219.37 last 24.72) - open/close 6249 (time 1770687897.60 total 229.86 last 20.02) - open/close 6527 (time 1770687908.61 total 240.87 last 25.25) - open/close 6762 (time 1770687919.30 total 251.56 last 21.99) - open/close 7101 (time 1770687930.17 total 262.44 last 31.16) - open/close 7559 (time 1770687940.54 total 272.80 last 44.17) - open/close 7843 (time 1770687951.29 total 283.55 last 26.42) - open/close 7953 (time 1770687961.30 total 293.56 last 11.00) - open/close 8090 (time 1770687971.45 total 303.72 last 13.49) - open/close 8331 (time 1770687982.33 total 314.60 last 22.15) - open/close 8732 (time 1770687992.34 total 324.60 last 40.09) - open/close 8912 (time 1770688002.82 total 335.08 last 17.18) - open/close 9105 (time 1770688013.51 total 345.77 last 18.04) - open/close 9479 (time 1770688024.44 total 356.71 last 34.21) - open/close 9743 (time 1770688034.82 total 367.08 last 25.45) - open/close 9967 (time 1770688045.52 total 377.78 last 20.93) - open/close 10212 (time 1770688055.55 total 387.81 last 24.42) - open/close 10516 (time 1770688066.30 total 398.56 last 28.27) - open/close 10753 (time 1770688076.31 total 408.57 last 23.68) - open/close 12003 (time 1770688086.31 total 418.57 last 125.00) - open/close 13774 (time 1770688096.31 total 428.57 last 177.05) - open/close 15761 (time 1770688106.32 total 438.58 last 198.66) - open/close 17548 (time 1770688116.32 total 448.58 last 178.61) - open/close 20000 (time 1770688125.90 total 458.17 last 255.85) - open/close 22509 (time 1770688135.91 total 468.17 last 250.79) - open/close 24363 (time 1770688145.91 total 478.18 last 185.29) - open/close 26538 (time 1770688155.92 total 488.18 last 217.47) - open/close 28752 (time 1770688165.92 total 498.18 last 221.40) - open/close 30000 (time 1770688172.53 total 504.79 last 188.59) - open/close 31598 (time 1770688182.54 total 514.80 last 159.75) - open/close 33390 (time 1770688192.54 total 524.80 last 179.19) - open/close 35527 (time 1770688202.54 total 534.80 last 213.58) - open/close 37347 (time 1770688212.54 total 544.80 last 182.00) - open/close 39480 (time 1770688222.55 total 554.81 last 213.23) - open/close 40000 (time 1770688224.62 total 556.88 last 250.37) - open/close 42273 (time 1770688234.62 total 566.89 last 227.28) - open/close 44499 (time 1770688244.63 total 576.89 last 222.56) - open/close 47069 (time 1770688254.63 total 586.89 last 256.93) - open/close 49095 (time 1770688264.63 total 596.89 last 202.52) total: 50000 open/close in 601.49 seconds: 83.13 ops/second - unlinked 0 (time 1770688270 ; total 0 ; last 0) - unlinked 10000 (time 1770688298 ; total 28 ; last 28) - unlinked 20000 (time 1770688322 ; total 52 ; last 24) - unlinked 30000 (time 1770688346 ; total 76 ; last 24) - unlinked 40000 (time 1770688367 ; total 97 ; last 21) total: 50000 unlinks in 125 seconds: 400.000000 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg255-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg255-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg255-server stop mds service on oleg255-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg255-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg255-server' oleg255-server: oleg255-server.virtnet: executing unload_modules_local oleg255-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg255-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg255-client: oleg255-client: ssh exited with exit code 2 pdsh@oleg255-client: oleg255-server: ssh exited with exit code 2 pdsh@oleg255-client: oleg255-client: ssh exited with exit code 2 pdsh@oleg255-client: oleg255-server: ssh exited with exit code 2