-----============= acceptance-small: conf-sanity ============----- Tue Oct 7 09:12:07 EDT 2025 oleg216-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex': No such file or directory excepting tests: 32newtarball 110 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a Stopping clients: oleg216-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg216-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg216-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg216-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg216-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg216-server oleg216-server: oleg216-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg216-server' oleg216-server: oleg216-server.virtnet: executing load_modules_local oleg216-server: Loading modules from /home/green/git/lustre-release/lustre oleg216-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg216-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg216-server: detected 4 online CPUs by sysfs oleg216-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg216-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg216-server: oleg216-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg216-client: oleg216-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg216-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg216-server: oleg216-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg216-client: oleg216-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg216-server: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg216-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg216-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg216-server stop mds service on oleg216-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg216-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 09:13:28 (1759842808) Stopping clients: oleg216-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg216-client.virtnet /mnt/lustre2 (opts:-f) oleg216-server: oleg216-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg216-server' oleg216-server: oleg216-server.virtnet: executing load_modules_local oleg216-server: Loading modules from /home/green/git/lustre-release/lustre oleg216-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg216-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg216-server: detected 4 online CPUs by sysfs oleg216-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg216-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg216-server: oleg216-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg216-client: oleg216-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg216-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg216-server: oleg216-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg216-client: oleg216-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg216-client.virtnet: -o user_xattr,flock oleg216-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 16166 358 15808 3% /mnt/lustre[OST:0] filesystem_summary: 16137 329 15808 3% /mnt/lustre fail_loc=0x1312 fail_val=5 - open/close 4797 (time 1759842855.07 total 10.00 last 479.59) - open/close 8812 (time 1759842865.08 total 20.00 last 401.43) - open/close 10000 (time 1759842867.66 total 22.59 last 459.26) - open/close 14231 (time 1759842877.66 total 32.59 last 423.08) - open/close 18886 (time 1759842887.66 total 42.59 last 465.42) - open/close 20000 (time 1759842889.90 total 44.83 last 498.77) - open/close 24770 (time 1759842899.90 total 54.83 last 476.93) - open/close 28771 (time 1759842909.90 total 64.83 last 400.04) - open/close 30000 (time 1759842913.34 total 68.27 last 357.02) - open/close 34389 (time 1759842923.34 total 78.27 last 438.84) - open/close 39009 (time 1759842933.35 total 88.27 last 461.95) - open/close 43920 (time 1759842943.35 total 98.28 last 491.02) - open/close 48359 (time 1759842953.35 total 108.28 last 443.88) - open/close 50000 (time 1759842957.25 total 112.18 last 420.08) - open/close 54602 (time 1759842967.26 total 122.18 last 460.16) - open/close 59108 (time 1759842977.26 total 132.18 last 450.56) - open/close 63549 (time 1759842987.26 total 142.19 last 444.04) total: 64768 open/close in 144.89 seconds: 447.01 ops/second - /unlink 6596 (time 1759843001.25 total 10.00 last 659.56) - /unlink 10000 (time 1759843006.94 total 15.68 last 599.07) - /unlink 16826 (time 1759843016.94 total 25.68 last 682.55) - /unlink 20000 (time 1759843021.52 total 30.26 last 692.88) - /unlink 27401 (time 1759843031.52 total 40.26 last 740.08) - /unlink 30000 (time 1759843035.19 total 43.94 last 707.24) - /unlink 36759 (time 1759843045.19 total 53.94 last 675.88) - /unlink 40000 (time 1759843049.65 total 58.40 last 727.30) - /unlink 47015 (time 1759843059.65 total 68.40 last 701.48) - /unlink 50000 (time 1759843063.55 total 72.30 last 764.46) - /unlink 58156 (time 1759843073.56 total 82.30 last 815.52) - /unlink 60000 (time 1759843075.91 total 84.65 last 784.26) total: 64768 /unlink in 90.65 seconds: 714.47 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4623 (time 1759843114.41 total 10.00 last 462.20) - open/close 8930 (time 1759843124.42 total 20.00 last 430.58) - open/close 10000 (time 1759843126.71 total 22.30 last 466.25) - open/close 14661 (time 1759843136.71 total 32.30 last 466.02) - open/close 18966 (time 1759843146.71 total 42.30 last 430.43) - open/close 20000 (time 1759843149.03 total 44.61 last 447.36) - open/close 24239 (time 1759843159.03 total 54.62 last 423.86) - open/close 28302 (time 1759843169.03 total 64.62 last 406.23) - open/close 30000 (time 1759843173.02 total 68.61 last 424.86) - open/close 34360 (time 1759843183.02 total 78.61 last 435.99) - open/close 38741 (time 1759843193.03 total 88.62 last 438.01) - open/close 40000 (time 1759843196.13 total 91.72 last 405.24) - open/close 44174 (time 1759843206.13 total 101.72 last 417.37) - open/close 48368 (time 1759843216.13 total 111.72 last 419.39) - open/close 50000 (time 1759843219.91 total 115.50 last 431.96) - open/close 54289 (time 1759843229.91 total 125.50 last 428.85) - open/close 58565 (time 1759843239.92 total 135.50 last 427.52) - open/close 60000 (time 1759843243.33 total 138.92 last 420.74) - open/close 64179 (time 1759843253.33 total 148.92 last 417.81) total: 64768 open/close in 150.43 seconds: 430.56 ops/second - /unlink 6653 (time 1759843266.28 total 10.00 last 665.29) - /unlink 10000 (time 1759843271.24 total 14.96 last 674.98) - /unlink 17008 (time 1759843281.24 total 24.96 last 700.77) - /unlink 20000 (time 1759843285.33 total 29.05 last 732.10) - /unlink 26924 (time 1759843295.36 total 39.08 last 689.87) - /unlink 30000 (time 1759843299.68 total 43.40 last 712.85) - /unlink 37191 (time 1759843309.68 total 53.40 last 719.02) - /unlink 40000 (time 1759843313.78 total 57.50 last 684.79) - /unlink 46978 (time 1759843323.78 total 67.50 last 697.79) - /unlink 50000 (time 1759843328.08 total 71.79 last 703.88) - /unlink 57030 (time 1759843338.08 total 81.80 last 702.95) - /unlink 60000 (time 1759843342.32 total 86.03 last 700.60) total: 64768 /unlink in 92.42 seconds: 700.81 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4666 (time 1759843380.95 total 10.00 last 466.56) - open/close 9290 (time 1759843390.95 total 20.00 last 462.37) - open/close 13826 (time 1759843400.95 total 30.00 last 453.59) - open/close 18523 (time 1759843410.96 total 40.00 last 469.62) - open/close 20000 (time 1759843414.09 total 43.13 last 472.05) - open/close 24507 (time 1759843424.09 total 53.13 last 450.61) - open/close 28962 (time 1759843434.09 total 63.13 last 445.48) - open/close 30000 (time 1759843436.57 total 65.61 last 418.99) - open/close 34477 (time 1759843446.57 total 75.61 last 447.63) - open/close 39033 (time 1759843456.57 total 85.62 last 455.49) - open/close 40000 (time 1759843458.71 total 87.75 last 452.54) - open/close 44535 (time 1759843468.71 total 97.75 last 453.42) - open/close 48756 (time 1759843478.71 total 107.76 last 422.04) - open/close 50000 (time 1759843481.56 total 110.61 last 435.85) - open/close 54251 (time 1759843491.56 total 120.61 last 425.06) - open/close 58662 (time 1759843501.57 total 130.61 last 441.03) - open/close 60000 (time 1759843504.60 total 133.64 last 441.31) - open/close 64408 (time 1759843514.60 total 143.65 last 440.73) total: 64768 open/close in 144.56 seconds: 448.03 ops/second - /unlink 7299 (time 1759843526.90 total 10.00 last 729.85) - /unlink 10000 (time 1759843530.23 total 13.33 last 812.08) - /unlink 17266 (time 1759843540.23 total 23.33 last 726.58) - /unlink 20000 (time 1759843544.19 total 27.28 last 690.85) - /unlink 26012 (time 1759843554.19 total 37.28 last 601.19) - /unlink 30000 (time 1759843560.23 total 43.33 last 659.66) - /unlink 36892 (time 1759843570.23 total 53.33 last 689.16) - /unlink 40000 (time 1759843574.75 total 57.84 last 688.50) - /unlink 47072 (time 1759843584.75 total 67.85 last 707.13) - /unlink 50000 (time 1759843588.87 total 71.97 last 710.25) - /unlink 56694 (time 1759843598.87 total 81.97 last 669.33) - /unlink 60000 (time 1759843603.48 total 86.58 last 716.81) total: 64768 /unlink in 93.14 seconds: 695.41 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2998 (time 1759843641.77 total 10.01 last 299.65) - open/close 4217 (time 1759843651.77 total 20.01 last 121.87) - open/close 5307 (time 1759843661.80 total 30.04 last 108.70) - open/close 8790 (time 1759843671.80 total 40.04 last 348.22) - open/close 10000 (time 1759843674.51 total 42.74 last 447.66) - open/close 14792 (time 1759843684.51 total 52.74 last 479.19) - open/close 19953 (time 1759843694.51 total 62.74 last 516.00) - open/close 24607 (time 1759843704.51 total 72.74 last 465.38) - open/close 29488 (time 1759843714.51 total 82.74 last 488.08) - open/close 34481 (time 1759843724.51 total 92.75 last 499.19) - open/close 39070 (time 1759843734.55 total 102.78 last 457.13) - open/close 40000 (time 1759843736.65 total 104.88 last 443.73) - open/close 44274 (time 1759843746.65 total 114.88 last 427.32) - open/close 48280 (time 1759843756.65 total 124.88 last 400.54) - open/close 50000 (time 1759843760.80 total 129.03 last 414.54) - open/close 54327 (time 1759843770.80 total 139.03 last 432.67) - open/close 58768 (time 1759843780.80 total 149.03 last 444.08) - open/close 60000 (time 1759843783.35 total 151.59 last 482.78) total: 64768 open/close in 161.46 seconds: 401.14 ops/second - /unlink 6850 (time 1759843804.64 total 10.00 last 684.93) - /unlink 10000 (time 1759843808.68 total 14.04 last 780.54) - /unlink 16822 (time 1759843818.68 total 24.04 last 682.11) - /unlink 20000 (time 1759843823.34 total 28.70 last 681.53) - /unlink 26905 (time 1759843833.35 total 38.70 last 690.36) - /unlink 30000 (time 1759843838.12 total 43.48 last 647.73) - /unlink 37130 (time 1759843848.12 total 53.48 last 712.97) - /unlink 40000 (time 1759843853.78 total 59.14 last 507.18) - /unlink 45337 (time 1759843863.78 total 69.14 last 533.63) - /unlink 50000 (time 1759843872.20 total 77.55 last 554.34) - /unlink 56701 (time 1759843882.20 total 87.55 last 670.09) - /unlink 60000 (time 1759843887.31 total 92.67 last 645.04) total: 64768 /unlink in 99.37 seconds: 651.78 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4018 (time 1759843922.68 total 10.00 last 401.76) - open/close 7470 (time 1759843932.68 total 20.00 last 345.12) - open/close 10000 (time 1759843940.13 total 27.45 last 339.54) - open/close 13457 (time 1759843950.13 total 37.45 last 345.70) - open/close 17328 (time 1759843960.13 total 47.45 last 387.09) - open/close 20000 (time 1759843967.39 total 54.72 last 367.89) - open/close 23641 (time 1759843977.40 total 64.72 last 364.02) - open/close 27209 (time 1759843987.40 total 74.72 last 356.74) - open/close 30000 (time 1759843993.95 total 81.27 last 426.02) - open/close 34419 (time 1759844003.95 total 91.27 last 441.88) - open/close 39076 (time 1759844013.95 total 101.27 last 465.63) - open/close 40000 (time 1759844015.97 total 103.29 last 458.14) - open/close 44633 (time 1759844025.97 total 113.29 last 463.25) - open/close 48763 (time 1759844035.97 total 123.29 last 412.93) - open/close 50000 (time 1759844039.31 total 126.63 last 370.77) - open/close 53477 (time 1759844049.31 total 136.63 last 347.67) - open/close 57887 (time 1759844059.31 total 146.63 last 440.95) - open/close 60000 (time 1759844064.09 total 151.42 last 441.78) - open/close 64324 (time 1759844074.09 total 161.42 last 432.39) total: 64768 open/close in 162.48 seconds: 398.62 ops/second - /unlink 7118 (time 1759844086.44 total 10.00 last 711.80) - /unlink 10000 (time 1759844090.36 total 13.92 last 735.27) - /unlink 17416 (time 1759844100.36 total 23.92 last 741.52) - /unlink 20000 (time 1759844103.63 total 27.20 last 788.42) - /unlink 27489 (time 1759844113.64 total 37.20 last 748.83) - /unlink 30000 (time 1759844117.18 total 40.74 last 708.42) - /unlink 37057 (time 1759844127.18 total 50.74 last 705.68) - /unlink 40000 (time 1759844131.92 total 55.49 last 620.58) - /unlink 46748 (time 1759844141.92 total 65.49 last 674.79) - /unlink 50000 (time 1759844146.30 total 69.86 last 743.10) - /unlink 56456 (time 1759844156.30 total 79.86 last 645.56) - /unlink 60000 (time 1759844161.38 total 84.94 last 697.72) total: 64768 /unlink in 91.79 seconds: 705.61 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg216-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg216-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg216-server stop mds service on oleg216-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg216-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg216-server' oleg216-server: oleg216-server.virtnet: executing unload_modules_local oleg216-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg216-server: LNET ready to unload oleg216-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. pdsh@oleg216-client: oleg216-client: ssh exited with exit code 2 pdsh@oleg216-client: oleg216-server: ssh exited with exit code 2 pdsh@oleg216-client: oleg216-server: ssh exited with exit code 2 pdsh@oleg216-client: oleg216-client: ssh exited with exit code 2 PASS 106 (1399s) Stopping clients: oleg216-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg216-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg216-client: oleg216-server: ssh exited with exit code 2 oleg216-server: oleg216-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg216-server' oleg216-server: oleg216-server.virtnet: executing load_modules_local oleg216-server: Loading modules from /home/green/git/lustre-release/lustre oleg216-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg216-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg216-server: detected 4 online CPUs by sysfs oleg216-server: Force libcfs to create 2 CPU partitions oleg216-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg216-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 1505 sec ========== 09:37:13 (1759844233)