== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 09:39:37 (1786714777) Stopping clients: oleg252-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg252-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg252-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg252-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg252-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg252-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg252-server oleg252-server: oleg252-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg252-server' oleg252-server: oleg252-server.virtnet: executing load_modules_local oleg252-server: Loading modules from /home/green/git/lustre-release/lustre oleg252-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg252-server: detected 4 online CPUs by sysfs oleg252-server: MODOPTS_LIBCFS= oleg252-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg252-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg252-server: oleg252-server.virtnet: executing set_default_debug -1 all pdsh@oleg252-client: oleg252-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg252-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg252-server: oleg252-server.virtnet: executing set_default_debug -1 all pdsh@oleg252-client: oleg252-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg252-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg252-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg252-server: oleg252-server.virtnet: executing set_default_debug -1 all pdsh@oleg252-client: oleg252-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg252-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg252-client.virtnet: -o user_xattr,flock 192.168.202.152@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush2 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 744 (time 1786715009.08 total 10.00 last 74.38) - open/close 1404 (time 1786715019.09 total 20.01 last 65.94) - open/close 2175 (time 1786715029.10 total 30.02 last 77.01) - open/close 2879 (time 1786715039.12 total 40.04 last 70.29) - open/close 3476 (time 1786715049.14 total 50.06 last 59.58) - open/close 4061 (time 1786715059.18 total 60.10 last 58.26) total: 4500 open/close in 67.31 seconds: 66.86 ops/second - /unlink 1370 (time 1786715082.39 total 10.00 last 136.97) - /unlink 2667 (time 1786715092.39 total 20.00 last 129.68) - /unlink 3915 (time 1786715102.40 total 30.01 last 124.78) total: 4500 /unlink in 34.30 seconds: 131.21 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 710 (time 1786715121.17 total 10.02 last 70.88) - open/close 1382 (time 1786715131.17 total 20.02 last 67.16) - open/close 2132 (time 1786715141.17 total 30.02 last 74.99) - open/close 2824 (time 1786715151.19 total 40.04 last 69.10) - open/close 3364 (time 1786715161.19 total 50.04 last 53.98) - open/close 3917 (time 1786715171.20 total 60.05 last 55.24) - open/close 4455 (time 1786715181.21 total 70.06 last 53.76) total: 4500 open/close in 70.59 seconds: 63.75 ops/second - /unlink 1238 (time 1786715197.68 total 10.01 last 123.71) - /unlink 2638 (time 1786715207.68 total 20.01 last 139.96) - /unlink 3746 (time 1786715217.68 total 30.01 last 110.79) total: 4500 /unlink in 35.73 seconds: 125.96 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 719 (time 1786715238.28 total 10.00 last 71.89) - open/close 1492 (time 1786715248.29 total 20.01 last 77.24) - open/close 2319 (time 1786715258.29 total 30.01 last 82.69) - open/close 3003 (time 1786715268.31 total 40.04 last 68.23) - open/close 3608 (time 1786715278.32 total 50.04 last 60.47) - open/close 4381 (time 1786715288.32 total 60.05 last 77.26) total: 4500 open/close in 62.24 seconds: 72.30 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1286 (time 1786715305.75 total 10.02 last 128.41) - /unlink 2274 (time 1786715315.77 total 20.03 last 98.65) - /unlink 3312 (time 1786715325.77 total 30.03 last 103.77) - /unlink 4446 (time 1786715335.77 total 40.04 last 113.36) total: 4500 /unlink in 40.78 seconds: 110.34 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 702 (time 1786715352.75 total 10.01 last 70.16) - open/close 1393 (time 1786715362.75 total 20.01 last 69.06) - open/close 2066 (time 1786715372.76 total 30.02 last 67.25) - open/close 2904 (time 1786715382.76 total 40.02 last 83.74) - open/close 3730 (time 1786715392.77 total 50.03 last 82.57) - open/close 4351 (time 1786715402.77 total 60.03 last 62.10) total: 4500 open/close in 63.47 seconds: 70.90 ops/second - /unlink 1169 (time 1786715422.90 total 10.01 last 116.73) - /unlink 2195 (time 1786715432.90 total 20.02 last 102.53) - /unlink 3571 (time 1786715442.91 total 30.03 last 137.49) - /unlink 4482 (time 1786715452.92 total 40.04 last 91.01) total: 4500 /unlink in 40.26 seconds: 111.78 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 785 (time 1786715469.84 total 10.01 last 78.42) - open/close 1493 (time 1786715479.87 total 20.04 last 70.62) - open/close 2143 (time 1786715489.88 total 30.04 last 64.94) - open/close 2818 (time 1786715499.88 total 40.05 last 67.48) - open/close 3460 (time 1786715509.89 total 50.05 last 64.18) - open/close 4167 (time 1786715519.89 total 60.06 last 70.66) total: 4500 open/close in 64.64 seconds: 69.62 ops/second - /unlink 1362 (time 1786715539.47 total 10.00 last 136.16) - /unlink 2781 (time 1786715549.47 total 20.00 last 141.87) - /unlink 4111 (time 1786715559.47 total 30.01 last 132.98) total: 4500 /unlink in 33.03 seconds: 136.24 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 683 (time 1786715577.46 total 10.01 last 68.20) - open/close 1292 (time 1786715587.49 total 20.04 last 60.75) - open/close 1804 (time 1786715597.49 total 30.04 last 51.18) - open/close 2589 (time 1786715607.53 total 40.08 last 78.21) - open/close 3284 (time 1786715617.53 total 50.08 last 69.48) - open/close 3905 (time 1786715627.56 total 60.12 last 61.89) - open/close 4387 (time 1786715637.58 total 70.14 last 48.11) total: 4500 open/close in 72.08 seconds: 62.43 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1243 (time 1786715656.98 total 10.01 last 124.18) - /unlink 2569 (time 1786715666.99 total 20.01 last 132.57) - /unlink 3556 (time 1786715677.01 total 30.04 last 98.47) total: 4500 /unlink in 39.61 seconds: 113.60 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 722 (time 1786715702.37 total 10.01 last 72.15) - open/close 1317 (time 1786715712.38 total 20.01 last 59.46) - open/close 1983 (time 1786715722.38 total 30.02 last 66.55) - open/close 2697 (time 1786715732.39 total 40.02 last 71.38) - open/close 3276 (time 1786715742.40 total 50.03 last 57.85) - open/close 4005 (time 1786715752.40 total 60.04 last 72.88) total: 4500 open/close in 68.38 seconds: 65.81 ops/second - /unlink 1294 (time 1786715776.72 total 10.00 last 129.38) - /unlink 2414 (time 1786715786.72 total 20.01 last 111.96) - /unlink 3588 (time 1786715796.73 total 30.01 last 117.35) total: 4500 /unlink in 38.51 seconds: 116.84 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 613 (time 1786715820.96 total 10.01 last 61.25) - open/close 1222 (time 1786715830.96 total 20.01 last 60.90) - open/close 1808 (time 1786715840.97 total 30.02 last 58.53) - open/close 2360 (time 1786715850.99 total 40.04 last 55.08) - open/close 3038 (time 1786715861.00 total 50.05 last 67.74) - open/close 3748 (time 1786715871.00 total 60.05 last 71.00) - open/close 4445 (time 1786715881.00 total 70.05 last 69.70) total: 4500 open/close in 70.77 seconds: 63.59 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1200 (time 1786715896.59 total 10.00 last 119.98) - /unlink 2260 (time 1786715906.60 total 20.00 last 105.97) - /unlink 3170 (time 1786715916.60 total 30.01 last 90.96) - /unlink 4091 (time 1786715926.61 total 40.02 last 92.05) total: 4500 /unlink in 44.41 seconds: 101.32 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 591 (time 1786715947.49 total 10.00 last 59.09) - open/close 1235 (time 1786715957.53 total 20.05 last 64.11) - open/close 1943 (time 1786715967.54 total 30.05 last 70.76) - open/close 2538 (time 1786715977.54 total 40.06 last 59.48) - open/close 3183 (time 1786715987.54 total 50.06 last 64.49) - open/close 3691 (time 1786715997.56 total 60.08 last 50.70) - open/close 4162 (time 1786716007.57 total 70.09 last 47.06) total: 4500 open/close in 75.77 seconds: 59.39 ops/second - /unlink 1210 (time 1786716028.26 total 10.00 last 120.95) - /unlink 2028 (time 1786716038.26 total 20.01 last 81.78) - /unlink 3147 (time 1786716048.27 total 30.01 last 111.81) - /unlink 4185 (time 1786716058.27 total 40.02 last 103.78) total: 4500 /unlink in 43.24 seconds: 104.08 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 646 (time 1786716077.56 total 10.02 last 64.45) - open/close 1313 (time 1786716087.56 total 20.03 last 66.68) - open/close 2088 (time 1786716097.57 total 30.03 last 77.47) - open/close 2780 (time 1786716107.57 total 40.03 last 69.19) - open/close 3448 (time 1786716117.57 total 50.03 last 66.80) - open/close 4200 (time 1786716127.57 total 60.03 last 75.19) total: 4500 open/close in 64.18 seconds: 70.12 ops/second - /unlink 1263 (time 1786716147.65 total 10.01 last 126.17) - /unlink 2262 (time 1786716157.66 total 20.01 last 99.85) - /unlink 3568 (time 1786716167.66 total 30.02 last 130.57) total: 4500 /unlink in 37.20 seconds: 120.96 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 776 (time 1786716189.67 total 10.01 last 77.51) - open/close 1453 (time 1786716199.68 total 20.02 last 67.66) - open/close 2129 (time 1786716209.68 total 30.02 last 67.60) - open/close 2903 (time 1786716219.69 total 40.03 last 77.31) - open/close 3577 (time 1786716229.69 total 50.03 last 67.37) - open/close 4296 (time 1786716239.69 total 60.04 last 71.89) total: 4500 open/close in 62.62 seconds: 71.86 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1379 (time 1786716256.91 total 10.01 last 137.80) - /unlink 2405 (time 1786716266.91 total 20.01 last 102.55) - /unlink 3715 (time 1786716276.92 total 30.01 last 130.96) total: 4500 /unlink in 37.00 seconds: 121.61 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 563 (time 1786716299.99 total 10.01 last 56.24) - open/close 1165 (time 1786716310.01 total 20.03 last 60.10) - open/close 1701 (time 1786716320.01 total 30.03 last 53.57) - open/close 2353 (time 1786716330.02 total 40.04 last 65.13) - open/close 3043 (time 1786716340.04 total 50.06 last 68.89) - open/close 3745 (time 1786716350.05 total 60.08 last 70.09) - open/close 4400 (time 1786716360.07 total 70.09 last 65.38) total: 4500 open/close in 71.61 seconds: 62.84 ops/second - /unlink 1344 (time 1786716377.22 total 10.01 last 134.29) - /unlink 2645 (time 1786716387.23 total 20.01 last 130.05) - /unlink 3952 (time 1786716397.23 total 30.01 last 130.68) total: 4500 /unlink in 34.67 seconds: 129.81 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 716 (time 1786716417.49 total 10.00 last 71.58) - open/close 1419 (time 1786716427.49 total 20.00 last 70.30) - open/close 2108 (time 1786716437.50 total 30.01 last 68.85) - open/close 2650 (time 1786716447.51 total 40.02 last 54.15) - open/close 3214 (time 1786716457.51 total 50.02 last 56.39) - open/close 3749 (time 1786716467.51 total 60.02 last 53.48) - open/close 4377 (time 1786716477.51 total 70.03 last 62.78) total: 4500 open/close in 71.95 seconds: 62.55 ops/second - /unlink 1204 (time 1786716495.13 total 10.00 last 120.38) - /unlink 2206 (time 1786716505.13 total 20.00 last 100.18) - /unlink 3565 (time 1786716515.13 total 30.01 last 135.84) total: 4500 /unlink in 37.91 seconds: 118.70 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 687 (time 1786716538.71 total 10.02 last 68.53) - open/close 1390 (time 1786716548.73 total 20.05 last 70.15) - open/close 1947 (time 1786716558.75 total 30.07 last 55.59) - open/close 2602 (time 1786716568.77 total 40.09 last 65.35) - open/close 3338 (time 1786716578.77 total 50.09 last 73.59) - open/close 4143 (time 1786716588.78 total 60.09 last 80.49) total: 4500 open/close in 65.06 seconds: 69.17 ops/second - /unlink 1451 (time 1786716608.47 total 10.03 last 144.69) - /unlink 2462 (time 1786716618.49 total 20.05 last 100.92) - /unlink 3387 (time 1786716628.49 total 30.05 last 92.47) total: 4500 /unlink in 39.52 seconds: 113.87 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 695 (time 1786716652.83 total 10.01 last 69.44) - open/close 1437 (time 1786716662.83 total 20.01 last 74.17) - open/close 2139 (time 1786716672.84 total 30.02 last 70.18) - open/close 2783 (time 1786716682.85 total 40.02 last 64.34) - open/close 3471 (time 1786716692.85 total 50.03 last 68.75) - open/close 4268 (time 1786716702.86 total 60.04 last 79.61) total: 4500 open/close in 63.61 seconds: 70.75 ops/second - /unlink 1158 (time 1786716721.35 total 10.00 last 115.79) - /unlink 2473 (time 1786716731.35 total 20.01 last 131.41) - /unlink 3828 (time 1786716741.36 total 30.01 last 135.47) total: 4500 /unlink in 37.15 seconds: 121.13 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 762 (time 1786716764.31 total 10.00 last 76.17) - open/close 1540 (time 1786716774.32 total 20.01 last 77.77) - open/close 2305 (time 1786716784.33 total 30.02 last 76.44) - open/close 3113 (time 1786716794.33 total 40.02 last 80.77) - open/close 3842 (time 1786716804.34 total 50.03 last 72.84) total: 4500 open/close in 59.20 seconds: 76.01 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1352 (time 1786716828.45 total 10.01 last 135.12) - /unlink 2566 (time 1786716838.45 total 20.01 last 121.36) - /unlink 3777 (time 1786716848.45 total 30.01 last 121.10) total: 4500 /unlink in 35.60 seconds: 126.41 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 692 (time 1786716869.24 total 10.03 last 68.98) - open/close 1266 (time 1786716879.25 total 20.04 last 57.34) - open/close 1982 (time 1786716889.26 total 30.05 last 71.54) - open/close 2667 (time 1786716899.27 total 40.06 last 68.41) - open/close 3324 (time 1786716909.28 total 50.07 last 65.66) - open/close 4058 (time 1786716919.29 total 60.08 last 73.34) total: 4500 open/close in 65.91 seconds: 68.27 ops/second - /unlink 1308 (time 1786716940.56 total 10.03 last 130.44) - /unlink 2573 (time 1786716950.57 total 20.03 last 126.42) - /unlink 3891 (time 1786716960.57 total 30.04 last 131.73) total: 4500 /unlink in 34.99 seconds: 128.62 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 781 (time 1786716980.63 total 10.01 last 78.02) - open/close 1400 (time 1786716990.65 total 20.04 last 61.74) - open/close 2122 (time 1786717000.66 total 30.04 last 72.17) - open/close 2827 (time 1786717010.67 total 40.06 last 70.40) - open/close 3599 (time 1786717020.67 total 50.06 last 77.20) - open/close 4288 (time 1786717030.68 total 60.07 last 68.82) total: 4500 open/close in 63.21 seconds: 71.19 ops/second - /unlink 988 (time 1786717048.85 total 10.00 last 98.79) - /unlink 2263 (time 1786717058.85 total 20.00 last 127.49) - /unlink 3488 (time 1786717068.86 total 30.01 last 122.37) total: 4500 /unlink in 38.61 seconds: 116.54 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11316: 95286 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg252-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg252-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg252-server stop mds service on oleg252-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg252-server stop mds service on oleg252-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg252-server unloading modules via unload_modules_local on: 'oleg252-server' oleg252-server: oleg252-server.virtnet: executing unload_modules_local oleg252-server: modules unloaded.