== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 15:01:54 (1779130914) Stopping clients: oleg222-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg222-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg222-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg222-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg222-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg222-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg222-server oleg222-server: oleg222-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1050: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg222-server' oleg222-server: oleg222-server.virtnet: executing load_modules_local oleg222-server: Loading modules from /home/green/git/lustre-release/lustre oleg222-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1050: echo: write error: Device or resource busy oleg222-server: detected 4 online CPUs by sysfs oleg222-server: MODOPTS_LIBCFS= oleg222-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg222-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg222-server: oleg222-server.virtnet: executing set_default_debug -1 all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg222-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg222-server: oleg222-server.virtnet: executing set_default_debug -1 all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg222-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg222-server: oleg222-server.virtnet: executing set_default_debug -1 all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg222-client.virtnet: -o user_xattr,flock 192.168.202.122@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H fnv_1a_64 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 572 (time 1779131119.49 total 10.01 last 57.13) - open/close 1137 (time 1779131129.51 total 20.03 last 56.39) - open/close 1760 (time 1779131139.52 total 30.04 last 62.25) - open/close 2451 (time 1779131149.55 total 40.07 last 68.92) - open/close 3195 (time 1779131159.55 total 50.07 last 74.40) - open/close 4080 (time 1779131169.55 total 60.07 last 88.44) total: 4500 open/close in 65.25 seconds: 68.97 ops/second - /unlink 1401 (time 1779131189.70 total 10.01 last 139.99) - /unlink 2787 (time 1779131199.70 total 20.01 last 138.54) - /unlink 4048 (time 1779131209.71 total 30.02 last 126.03) total: 4500 /unlink in 32.73 seconds: 137.47 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 745 (time 1779131226.64 total 10.01 last 74.41) - open/close 1690 (time 1779131236.64 total 20.02 last 94.44) - open/close 2571 (time 1779131246.65 total 30.02 last 88.07) - open/close 3465 (time 1779131256.65 total 40.03 last 89.35) - open/close 4189 (time 1779131266.66 total 50.03 last 72.39) total: 4500 open/close in 53.58 seconds: 83.98 ops/second - /unlink 1737 (time 1779131284.17 total 10.00 last 173.64) - /unlink 3456 (time 1779131294.17 total 20.01 last 171.79) total: 4500 /unlink in 27.05 seconds: 166.35 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 677 (time 1779131316.61 total 10.00 last 67.69) - open/close 1315 (time 1779131326.62 total 20.01 last 63.72) - open/close 2134 (time 1779131336.63 total 30.02 last 81.87) - open/close 3009 (time 1779131346.63 total 40.02 last 87.48) - open/close 3916 (time 1779131356.64 total 50.03 last 90.63) total: 4500 open/close in 57.93 seconds: 77.69 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1510 (time 1779131379.80 total 10.04 last 150.47) - /unlink 3126 (time 1779131389.80 total 20.04 last 161.59) total: 4500 /unlink in 28.11 seconds: 160.10 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 904 (time 1779131412.05 total 10.00 last 90.36) - open/close 1528 (time 1779131422.06 total 20.01 last 62.36) - open/close 2373 (time 1779131432.06 total 30.01 last 84.48) - open/close 3234 (time 1779131442.08 total 40.03 last 85.92) - open/close 4101 (time 1779131452.09 total 50.04 last 86.62) total: 4500 open/close in 55.44 seconds: 81.17 ops/second - /unlink 1262 (time 1779131472.05 total 10.00 last 126.16) - /unlink 2556 (time 1779131482.07 total 20.02 last 129.15) - /unlink 4069 (time 1779131492.08 total 30.03 last 151.18) total: 4500 /unlink in 33.34 seconds: 134.98 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 742 (time 1779131509.92 total 10.03 last 73.99) - open/close 1470 (time 1779131519.92 total 20.03 last 72.78) - open/close 2214 (time 1779131529.93 total 30.03 last 74.39) - open/close 2802 (time 1779131539.94 total 40.04 last 58.74) - open/close 3405 (time 1779131549.94 total 50.05 last 60.26) - open/close 4218 (time 1779131559.95 total 60.06 last 81.20) total: 4500 open/close in 63.32 seconds: 71.07 ops/second - /unlink 1412 (time 1779131576.97 total 10.01 last 140.99) - /unlink 2576 (time 1779131586.97 total 20.02 last 116.38) - /unlink 3893 (time 1779131596.98 total 30.02 last 131.66) total: 4500 /unlink in 34.03 seconds: 132.22 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 919 (time 1779131615.86 total 10.00 last 91.89) - open/close 1726 (time 1779131625.86 total 20.00 last 80.67) - open/close 2556 (time 1779131635.87 total 30.01 last 82.95) - open/close 3217 (time 1779131645.87 total 40.01 last 66.07) - open/close 4061 (time 1779131655.87 total 50.01 last 84.40) total: 4500 open/close in 55.58 seconds: 80.97 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1658 (time 1779131675.50 total 10.01 last 165.63) - /unlink 3181 (time 1779131685.50 total 20.01 last 152.24) total: 4500 /unlink in 28.35 seconds: 158.72 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 893 (time 1779131707.94 total 10.00 last 89.26) - open/close 1700 (time 1779131717.95 total 20.02 last 80.61) - open/close 2644 (time 1779131727.97 total 30.03 last 94.25) - open/close 3317 (time 1779131737.97 total 40.03 last 67.30) - open/close 4064 (time 1779131747.98 total 50.05 last 74.59) total: 4500 open/close in 57.44 seconds: 78.34 ops/second - /unlink 1372 (time 1779131772.75 total 10.00 last 137.17) - /unlink 2551 (time 1779131782.76 total 20.01 last 117.86) - /unlink 3769 (time 1779131792.76 total 30.01 last 121.75) total: 4500 /unlink in 35.63 seconds: 126.29 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 538 (time 1779131813.22 total 10.01 last 53.76) - open/close 1198 (time 1779131823.23 total 20.02 last 65.92) - open/close 1907 (time 1779131833.25 total 30.04 last 70.76) - open/close 2673 (time 1779131843.25 total 40.05 last 76.55) - open/close 3356 (time 1779131853.26 total 50.06 last 68.23) - open/close 4138 (time 1779131863.27 total 60.06 last 78.16) total: 4500 open/close in 66.02 seconds: 68.16 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1416 (time 1779131883.79 total 10.00 last 141.58) - /unlink 2515 (time 1779131893.80 total 20.01 last 109.85) - /unlink 3527 (time 1779131903.80 total 30.01 last 101.16) total: 4500 /unlink in 38.57 seconds: 116.68 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 834 (time 1779131928.03 total 10.00 last 83.37) - open/close 1637 (time 1779131938.04 total 20.02 last 80.20) - open/close 2384 (time 1779131948.04 total 30.02 last 74.69) - open/close 3053 (time 1779131958.06 total 40.03 last 66.79) - open/close 3666 (time 1779131968.06 total 50.03 last 61.30) - open/close 4233 (time 1779131978.09 total 60.06 last 56.54) total: 4500 open/close in 65.23 seconds: 68.99 ops/second - /unlink 1402 (time 1779131997.87 total 10.00 last 140.18) - /unlink 2733 (time 1779132007.88 total 20.01 last 133.04) - /unlink 4197 (time 1779132017.88 total 30.01 last 146.35) total: 4500 /unlink in 32.12 seconds: 140.12 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 712 (time 1779132034.83 total 10.02 last 71.09) - open/close 1400 (time 1779132044.84 total 20.03 last 68.73) - open/close 2168 (time 1779132054.85 total 30.03 last 76.76) - open/close 2796 (time 1779132064.86 total 40.04 last 62.73) - open/close 3515 (time 1779132074.88 total 50.06 last 71.77) - open/close 4179 (time 1779132084.89 total 60.08 last 66.30) total: 4500 open/close in 64.81 seconds: 69.44 ops/second - /unlink 1506 (time 1779132104.00 total 10.01 last 150.51) - /unlink 2869 (time 1779132114.01 total 20.01 last 136.28) - /unlink 4060 (time 1779132124.01 total 30.01 last 119.06) total: 4500 /unlink in 33.57 seconds: 134.03 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 600 (time 1779132143.56 total 10.00 last 59.99) - open/close 1352 (time 1779132153.57 total 20.00 last 75.18) - open/close 2079 (time 1779132163.57 total 30.01 last 72.65) - open/close 2666 (time 1779132173.57 total 40.01 last 58.69) - open/close 3193 (time 1779132183.58 total 50.02 last 52.67) - open/close 3754 (time 1779132193.58 total 60.02 last 56.08) - open/close 4255 (time 1779132203.59 total 70.03 last 50.05) total: 4500 open/close in 74.20 seconds: 60.64 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1520 (time 1779132222.39 total 10.00 last 151.96) - /unlink 2997 (time 1779132232.39 total 20.01 last 147.66) - /unlink 4405 (time 1779132242.40 total 30.01 last 140.69) total: 4500 /unlink in 30.92 seconds: 145.56 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 778 (time 1779132257.98 total 10.00 last 77.77) - open/close 1238 (time 1779132268.01 total 20.03 last 45.87) - open/close 2042 (time 1779132278.03 total 30.05 last 80.23) - open/close 2766 (time 1779132288.04 total 40.06 last 72.36) - open/close 3394 (time 1779132298.05 total 50.07 last 62.73) - open/close 4092 (time 1779132308.05 total 60.07 last 69.79) total: 4500 open/close in 64.75 seconds: 69.50 ops/second - /unlink 1088 (time 1779132328.83 total 10.00 last 108.77) - /unlink 2445 (time 1779132338.83 total 20.01 last 135.64) - /unlink 3821 (time 1779132348.83 total 30.01 last 137.59) total: 4500 /unlink in 35.87 seconds: 125.47 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 891 (time 1779132370.30 total 10.01 last 89.00) - open/close 1625 (time 1779132380.31 total 20.02 last 73.37) - open/close 2353 (time 1779132390.31 total 30.02 last 72.77) - open/close 3060 (time 1779132400.32 total 40.03 last 70.61) - open/close 3753 (time 1779132410.32 total 50.03 last 69.30) - open/close 4388 (time 1779132420.33 total 60.04 last 63.43) total: 4500 open/close in 61.34 seconds: 73.36 ops/second - /unlink 1346 (time 1779132436.22 total 10.01 last 134.46) - /unlink 2618 (time 1779132446.23 total 20.02 last 127.07) - /unlink 3953 (time 1779132456.24 total 30.03 last 133.36) total: 4500 /unlink in 33.82 seconds: 133.05 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 708 (time 1779132475.49 total 10.00 last 70.79) - open/close 1340 (time 1779132485.50 total 20.01 last 63.15) - open/close 1988 (time 1779132495.52 total 30.03 last 64.69) - open/close 2670 (time 1779132505.53 total 40.04 last 68.12) - open/close 3246 (time 1779132515.54 total 50.05 last 57.56) - open/close 3973 (time 1779132525.54 total 60.05 last 72.66) total: 4500 open/close in 67.71 seconds: 66.46 ops/second - /unlink 1141 (time 1779132548.74 total 10.02 last 113.87) - /unlink 2546 (time 1779132558.75 total 20.02 last 140.45) - /unlink 3661 (time 1779132568.75 total 30.02 last 111.49) total: 4500 /unlink in 36.94 seconds: 121.80 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 654 (time 1779132592.34 total 10.01 last 65.36) - open/close 1243 (time 1779132602.34 total 20.01 last 58.87) - open/close 2057 (time 1779132612.35 total 30.02 last 81.36) - open/close 2849 (time 1779132622.35 total 40.02 last 79.18) - open/close 3575 (time 1779132632.36 total 50.03 last 72.50) - open/close 4375 (time 1779132642.36 total 60.03 last 79.98) total: 4500 open/close in 61.61 seconds: 73.03 ops/second - /unlink 1375 (time 1779132658.49 total 10.02 last 137.28) - /unlink 2747 (time 1779132668.50 total 20.02 last 137.09) - /unlink 4145 (time 1779132678.50 total 30.03 last 139.71) total: 4500 /unlink in 32.26 seconds: 139.49 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 626 (time 1779132695.68 total 10.00 last 62.57) - open/close 1267 (time 1779132705.70 total 20.02 last 63.97) - open/close 1973 (time 1779132715.71 total 30.03 last 70.57) - open/close 2717 (time 1779132725.71 total 40.03 last 74.39) - open/close 3484 (time 1779132735.73 total 50.05 last 76.54) - open/close 4099 (time 1779132745.73 total 60.05 last 61.50) total: 4500 open/close in 65.17 seconds: 69.05 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1386 (time 1779132765.49 total 10.00 last 138.54) - /unlink 2793 (time 1779132775.49 total 20.01 last 140.67) total: 4500 /unlink in 29.62 seconds: 151.91 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 897 (time 1779132798.93 total 10.00 last 89.67) - open/close 1582 (time 1779132808.94 total 20.01 last 68.42) - open/close 2391 (time 1779132818.94 total 30.02 last 80.89) - open/close 3246 (time 1779132828.95 total 40.02 last 85.44) - open/close 4177 (time 1779132838.95 total 50.03 last 93.05) total: 4500 open/close in 53.54 seconds: 84.05 ops/second - /unlink 1694 (time 1779132856.55 total 10.01 last 169.27) - /unlink 3103 (time 1779132866.57 total 20.03 last 140.61) total: 4500 /unlink in 28.31 seconds: 158.97 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 732 (time 1779132891.29 total 10.01 last 73.15) - open/close 1260 (time 1779132901.31 total 20.03 last 52.66) - open/close 1755 (time 1779132911.32 total 30.04 last 49.45) - open/close 2453 (time 1779132921.33 total 40.05 last 69.73) - open/close 3305 (time 1779132931.33 total 50.05 last 85.20) - open/close 3969 (time 1779132941.34 total 60.06 last 66.34) total: 4500 open/close in 69.38 seconds: 64.86 ops/second - /unlink 1387 (time 1779132966.80 total 10.00 last 138.69) - /unlink 2589 (time 1779132976.81 total 20.00 last 120.17) - /unlink 3882 (time 1779132986.81 total 30.01 last 129.24) total: 4500 /unlink in 33.95 seconds: 132.55 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11217: 94693 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg222-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg222-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg222-server stop mds service on oleg222-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg222-server stop mds service on oleg222-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg222-server unloading modules via unload_modules_local on: 'oleg222-server' oleg222-server: oleg222-server.virtnet: executing unload_modules_local oleg222-server: modules unloaded.