-----============= acceptance-small: conf-sanity ============----- Mon Jun 30 17:30:42 EDT 2025 oleg241-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex: No such file or directory excepting tests: 32newtarball 110 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a Stopping clients: oleg241-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg241-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg241-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg241-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg241-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg241-server oleg241-server: oleg241-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg241-server' oleg241-server: oleg241-server.virtnet: executing load_modules_local oleg241-server: Loading modules from /home/green/git/lustre-release/lustre oleg241-server: detected 4 online CPUs by sysfs oleg241-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg241-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg241-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg241-server: oleg241-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg241-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg241-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg241-server stop mds service on oleg241-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg241-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 17:31:45 (1751319105) Stopping clients: oleg241-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg241-client.virtnet /mnt/lustre2 (opts:-f) oleg241-server: oleg241-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg241-server' oleg241-server: oleg241-server.virtnet: executing load_modules_local oleg241-server: Loading modules from /home/green/git/lustre-release/lustre oleg241-server: detected 4 online CPUs by sysfs oleg241-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg241-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg241-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg241-client.virtnet: -o user_xattr,flock oleg241-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 16131 291 15840 2% /mnt/lustre[OST:0] filesystem_summary: 16169 329 15840 3% /mnt/lustre fail_loc=0x1312 fail_val=5 - open/close 4925 (time 1751319140.74 total 10.00 last 492.50) - open/close 9741 (time 1751319150.74 total 20.00 last 481.60) - open/close 14775 (time 1751319160.74 total 30.00 last 503.32) - open/close 19545 (time 1751319170.74 total 40.00 last 476.98) - open/close 25323 (time 1751319180.74 total 50.00 last 577.74) - open/close 30000 (time 1751319189.06 total 58.32 last 562.63) - open/close 35985 (time 1751319199.06 total 68.32 last 598.42) - open/close 40000 (time 1751319206.12 total 75.39 last 568.06) - open/close 45505 (time 1751319216.13 total 85.39 last 550.45) - open/close 50000 (time 1751319223.66 total 92.92 last 596.85) - open/close 56036 (time 1751319233.66 total 102.92 last 603.54) - open/close 60000 (time 1751319240.42 total 109.68 last 586.02) total: 64768 open/close in 118.08 seconds: 548.50 ops/second - /unlink 10000 (time 1751319259.43 total 9.83 last 1017.00) - /unlink 20000 (time 1751319269.21 total 19.61 last 1023.22) - /unlink 30000 (time 1751319278.96 total 29.36 last 1025.69) - /unlink 40000 (time 1751319288.44 total 38.84 last 1054.46) - /unlink 50000 (time 1751319297.96 total 48.35 last 1051.05) - /unlink 60000 (time 1751319307.72 total 58.12 last 1023.62) total: 64768 /unlink in 62.78 seconds: 1031.59 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4998 (time 1751319360.02 total 10.05 last 497.31) - open/close 10000 (time 1751319369.94 total 19.96 last 504.66) - open/close 15144 (time 1751319379.94 total 29.96 last 514.36) - open/close 19954 (time 1751319389.94 total 39.96 last 480.96) - open/close 24983 (time 1751319399.94 total 49.96 last 502.87) - open/close 30000 (time 1751319409.83 total 59.86 last 507.07) - open/close 35051 (time 1751319419.83 total 69.86 last 505.06) - open/close 40000 (time 1751319429.63 total 79.66 last 504.99) - open/close 45084 (time 1751319439.64 total 89.66 last 508.32) - open/close 50000 (time 1751319449.45 total 99.48 last 500.64) - open/close 55246 (time 1751319459.46 total 109.48 last 524.57) - open/close 60000 (time 1751319468.78 total 118.81 last 509.75) total: 64768 open/close in 128.31 seconds: 504.79 ops/second - /unlink 8667 (time 1751319489.13 total 10.00 last 866.61) - /unlink 16938 (time 1751319499.13 total 20.00 last 827.07) - /unlink 20000 (time 1751319502.63 total 23.50 last 876.40) - /unlink 28526 (time 1751319512.63 total 33.50 last 852.59) - /unlink 37294 (time 1751319522.63 total 43.50 last 876.79) - /unlink 40000 (time 1751319525.77 total 46.64 last 861.02) - /unlink 48977 (time 1751319535.77 total 56.64 last 897.61) - /unlink 57923 (time 1751319545.78 total 66.64 last 894.14) - /unlink 60000 (time 1751319547.98 total 68.85 last 941.86) total: 64768 /unlink in 74.03 seconds: 874.94 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 5447 (time 1751319590.59 total 10.00 last 544.64) - open/close 10000 (time 1751319598.76 total 18.17 last 557.33) - open/close 15487 (time 1751319608.76 total 28.17 last 548.62) - open/close 20000 (time 1751319617.13 total 36.54 last 539.41) - open/close 25415 (time 1751319627.13 total 46.54 last 541.48) - open/close 30000 (time 1751319635.54 total 54.95 last 545.11) - open/close 35348 (time 1751319645.54 total 64.95 last 534.73) - open/close 40000 (time 1751319654.18 total 73.59 last 538.55) - open/close 44986 (time 1751319664.18 total 83.59 last 498.56) - open/close 49404 (time 1751319674.21 total 93.62 last 440.56) - open/close 54311 (time 1751319684.21 total 103.62 last 490.62) - open/close 58437 (time 1751319694.21 total 113.62 last 412.58) - open/close 60000 (time 1751319697.34 total 116.75 last 499.54) total: 64768 open/close in 126.07 seconds: 513.74 ops/second - /unlink 9160 (time 1751319717.52 total 10.00 last 915.96) - /unlink 18117 (time 1751319727.52 total 20.00 last 895.65) - /unlink 20000 (time 1751319729.65 total 22.13 last 883.24) - /unlink 28716 (time 1751319739.65 total 32.13 last 871.51) - /unlink 37558 (time 1751319749.65 total 42.14 last 884.11) - /unlink 40000 (time 1751319752.38 total 44.86 last 895.32) - /unlink 49001 (time 1751319762.38 total 54.86 last 900.05) - /unlink 58136 (time 1751319772.38 total 64.86 last 913.42) - /unlink 60000 (time 1751319774.42 total 66.90 last 914.68) total: 64768 /unlink in 71.98 seconds: 899.81 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 3206 (time 1751319821.23 total 10.00 last 320.48) - open/close 6403 (time 1751319831.23 total 20.01 last 319.65) - open/close 10000 (time 1751319839.17 total 27.95 last 452.76) - open/close 14740 (time 1751319849.17 total 37.95 last 473.95) - open/close 17788 (time 1751319859.18 total 47.95 last 304.74) - open/close 20000 (time 1751319866.78 total 55.56 last 290.74) - open/close 22896 (time 1751319876.79 total 65.56 last 289.54) - open/close 25860 (time 1751319886.79 total 75.56 last 296.34) - open/close 29161 (time 1751319896.79 total 85.57 last 330.09) - open/close 32272 (time 1751319906.79 total 95.57 last 311.00) - open/close 35434 (time 1751319916.80 total 105.57 last 316.09) - open/close 38529 (time 1751319926.80 total 115.57 last 309.49) - open/close 40000 (time 1751319932.12 total 120.90 last 276.33) - open/close 43309 (time 1751319942.12 total 130.90 last 330.86) - open/close 45845 (time 1751319952.12 total 140.90 last 253.56) - open/close 48422 (time 1751319962.12 total 150.90 last 257.69) - open/close 50000 (time 1751319968.74 total 157.52 last 238.38) - open/close 52827 (time 1751319978.74 total 167.52 last 282.70) - open/close 55263 (time 1751319988.74 total 177.52 last 243.58) - open/close 57717 (time 1751319998.75 total 187.52 last 245.33) - open/close 60000 (time 1751320008.21 total 196.99 last 241.10) - open/close 62495 (time 1751320018.22 total 206.99 last 249.42) total: 64768 open/close in 216.29 seconds: 299.45 ops/second - /unlink 4471 (time 1751320038.80 total 10.00 last 447.05) - /unlink 9518 (time 1751320048.80 total 20.00 last 504.60) - /unlink 14060 (time 1751320058.81 total 30.00 last 454.13) - /unlink 18633 (time 1751320068.81 total 40.01 last 457.20) - /unlink 20000 (time 1751320071.70 total 42.90 last 471.97) - /unlink 24740 (time 1751320081.71 total 52.91 last 473.92) - /unlink 29379 (time 1751320091.71 total 62.91 last 463.86) - /unlink 34404 (time 1751320101.71 total 72.91 last 502.46) - /unlink 39104 (time 1751320111.71 total 82.91 last 469.95) - /unlink 40000 (time 1751320113.74 total 84.94 last 441.86) - /unlink 44588 (time 1751320123.74 total 94.94 last 458.78) - /unlink 49173 (time 1751320133.74 total 104.94 last 458.47) - /unlink 54253 (time 1751320143.74 total 114.94 last 507.94) - /unlink 59516 (time 1751320153.74 total 124.94 last 526.30) - /unlink 64335 (time 1751320163.74 total 134.94 last 481.90) total: 64768 /unlink in 135.89 seconds: 476.61 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2525 (time 1751320197.13 total 10.00 last 252.43) - open/close 5027 (time 1751320207.14 total 20.00 last 250.17) - open/close 7579 (time 1751320217.14 total 30.00 last 255.20) - open/close 10000 (time 1751320227.13 total 40.00 last 242.26) - open/close 12760 (time 1751320237.13 total 50.00 last 275.98) - open/close 15219 (time 1751320247.13 total 60.00 last 245.84) - open/close 17654 (time 1751320257.13 total 70.00 last 243.48) - open/close 20000 (time 1751320266.79 total 79.65 last 243.02) - open/close 22507 (time 1751320276.79 total 89.66 last 250.65) - open/close 25056 (time 1751320286.79 total 99.66 last 254.87) - open/close 27477 (time 1751320296.79 total 109.66 last 242.08) - open/close 29952 (time 1751320306.79 total 119.66 last 247.45) - open/close 32451 (time 1751320316.80 total 129.66 last 249.83) - open/close 34839 (time 1751320326.80 total 139.67 last 238.73) - open/close 37174 (time 1751320336.80 total 149.67 last 233.43) - open/close 39595 (time 1751320346.80 total 159.67 last 242.02) - open/close 42066 (time 1751320356.81 total 169.67 last 247.06) - open/close 44489 (time 1751320366.81 total 179.68 last 242.27) - open/close 46977 (time 1751320376.81 total 189.68 last 248.72) - open/close 49362 (time 1751320386.81 total 199.68 last 238.46) - open/close 50000 (time 1751320389.44 total 202.30 last 243.29) - open/close 52337 (time 1751320399.44 total 212.31 last 233.65) - open/close 54771 (time 1751320409.44 total 222.31 last 243.34) - open/close 57164 (time 1751320419.44 total 232.31 last 239.26) - open/close 59562 (time 1751320429.44 total 242.31 last 239.77) - open/close 62014 (time 1751320439.45 total 252.31 last 245.12) - open/close 64367 (time 1751320449.45 total 262.32 last 235.27) total: 64768 open/close in 264.04 seconds: 245.30 ops/second - /unlink 4570 (time 1751320462.41 total 10.00 last 456.93) - /unlink 9187 (time 1751320472.41 total 20.00 last 461.65) - /unlink 13773 (time 1751320482.41 total 30.00 last 458.59) - /unlink 18279 (time 1751320492.41 total 40.00 last 450.59) - /unlink 20000 (time 1751320496.25 total 43.84 last 448.26) - /unlink 24574 (time 1751320506.26 total 53.84 last 457.29) - /unlink 29380 (time 1751320516.26 total 63.85 last 480.56) - /unlink 33856 (time 1751320526.26 total 73.85 last 447.50) - /unlink 38480 (time 1751320536.26 total 83.85 last 462.35) - /unlink 40000 (time 1751320539.43 total 87.02 last 479.97) - /unlink 44784 (time 1751320549.43 total 97.02 last 478.30) - /unlink 49407 (time 1751320559.43 total 107.02 last 462.29) - /unlink 54047 (time 1751320569.43 total 117.02 last 464.00) - /unlink 58767 (time 1751320579.43 total 127.02 last 472.00) - /unlink 60000 (time 1751320582.03 total 129.62 last 473.73) - /unlink 64646 (time 1751320592.03 total 139.62 last 464.54) total: 64768 /unlink in 139.90 seconds: 462.97 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg241-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg241-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg241-server stop mds service on oleg241-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg241-server LNET ready to unload unloading modules on: 'oleg241-server' oleg241-server: oleg241-server.virtnet: executing unload_modules_local oleg241-server: LNET ready to unload modules unloaded. pdsh@oleg241-client: oleg241-client: ssh exited with exit code 2 pdsh@oleg241-client: oleg241-server: ssh exited with exit code 2 pdsh@oleg241-client: oleg241-client: ssh exited with exit code 2 pdsh@oleg241-client: oleg241-server: ssh exited with exit code 2 PASS 106 (1525s) Stopping clients: oleg241-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg241-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg241-client: oleg241-server: ssh exited with exit code 2 oleg241-server: oleg241-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg241-server' oleg241-server: oleg241-server.virtnet: executing load_modules_local oleg241-server: Loading modules from /home/green/git/lustre-release/lustre oleg241-server: detected 4 online CPUs by sysfs oleg241-server: Force libcfs to create 2 CPU partitions oleg241-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg241-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 1611 sec ========== 17:57:34 (1751320654)