== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 02:36:16 (1775198176) Stopping clients: oleg226-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg226-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg226-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg226-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg226-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg226-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg226-server oleg226-server: oleg226-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg226-server' oleg226-server: oleg226-server.virtnet: executing load_modules_local oleg226-server: Loading modules from /home/green/git/lustre-release/lustre oleg226-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg226-server: detected 4 online CPUs by sysfs oleg226-server: MODOPTS_LIBCFS= oleg226-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg226-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg226-server: oleg226-server.virtnet: executing set_default_debug -1 all pdsh@oleg226-client: oleg226-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg226-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg226-server: oleg226-server.virtnet: executing set_default_debug -1 all pdsh@oleg226-client: oleg226-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg226-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg226-server: oleg226-server.virtnet: executing set_default_debug -1 all pdsh@oleg226-client: oleg226-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg226-client.virtnet: -o user_xattr,flock 192.168.202.126@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H all_char /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 853 (time 1775198384.40 total 10.01 last 85.23) - open/close 1758 (time 1775198394.41 total 20.02 last 90.40) - open/close 2711 (time 1775198404.42 total 30.03 last 95.23) - open/close 3635 (time 1775198414.42 total 40.03 last 92.40) total: 4500 open/close in 49.07 seconds: 91.70 ops/second - /unlink 1534 (time 1775198437.45 total 10.00 last 153.33) - /unlink 3187 (time 1775198447.45 total 20.01 last 165.25) total: 4500 /unlink in 27.61 seconds: 162.97 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 945 (time 1775198469.65 total 10.00 last 94.46) - open/close 1849 (time 1775198479.65 total 20.00 last 90.40) - open/close 2762 (time 1775198489.66 total 30.01 last 91.21) - open/close 3735 (time 1775198499.66 total 40.02 last 97.24) total: 4500 open/close in 47.57 seconds: 94.59 ops/second - /unlink 1580 (time 1775198521.33 total 10.00 last 157.98) - /unlink 2768 (time 1775198531.33 total 20.00 last 118.77) - /unlink 3876 (time 1775198541.34 total 30.01 last 110.72) total: 4500 /unlink in 34.35 seconds: 131.00 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 920 (time 1775198560.80 total 10.00 last 91.96) - open/close 1798 (time 1775198570.83 total 20.04 last 87.51) - open/close 2666 (time 1775198580.83 total 30.04 last 86.79) - open/close 3531 (time 1775198590.84 total 40.05 last 86.39) - open/close 4276 (time 1775198600.85 total 50.06 last 74.43) total: 4500 open/close in 52.35 seconds: 85.96 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1747 (time 1775198618.03 total 10.00 last 174.68) - /unlink 3171 (time 1775198628.03 total 20.00 last 142.39) - /unlink 4385 (time 1775198638.04 total 30.02 last 121.24) total: 4500 /unlink in 31.10 seconds: 144.68 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 506 (time 1775198655.76 total 10.01 last 50.53) - open/close 1088 (time 1775198665.77 total 20.02 last 58.15) - open/close 1850 (time 1775198675.77 total 30.02 last 76.19) - open/close 2695 (time 1775198685.79 total 40.04 last 84.33) - open/close 3580 (time 1775198695.79 total 50.05 last 88.48) - open/close 4377 (time 1775198705.81 total 60.07 last 79.55) total: 4500 open/close in 61.40 seconds: 73.29 ops/second - /unlink 1619 (time 1775198721.63 total 10.00 last 161.90) - /unlink 3237 (time 1775198731.63 total 20.00 last 161.79) total: 4500 /unlink in 28.67 seconds: 156.99 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 797 (time 1775198755.74 total 10.01 last 79.65) - open/close 1566 (time 1775198765.75 total 20.01 last 76.85) - open/close 2114 (time 1775198775.75 total 30.02 last 54.78) - open/close 2848 (time 1775198785.76 total 40.03 last 73.31) - open/close 3340 (time 1775198795.78 total 50.05 last 49.10) - open/close 4028 (time 1775198805.78 total 60.05 last 68.79) total: 4500 open/close in 66.59 seconds: 67.58 ops/second - /unlink 1510 (time 1775198827.06 total 10.00 last 150.98) - /unlink 2907 (time 1775198837.06 total 20.00 last 139.67) - /unlink 3897 (time 1775198847.06 total 30.00 last 99.00) total: 4500 /unlink in 36.66 seconds: 122.74 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 756 (time 1775198868.99 total 10.01 last 75.55) - open/close 1461 (time 1775198879.01 total 20.03 last 70.35) - open/close 2222 (time 1775198889.02 total 30.04 last 76.03) - open/close 3024 (time 1775198899.04 total 40.06 last 80.05) - open/close 3755 (time 1775198909.05 total 50.07 last 73.04) total: 4500 open/close in 59.16 seconds: 76.06 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1410 (time 1775198933.38 total 10.01 last 140.90) - /unlink 2808 (time 1775198943.39 total 20.02 last 139.66) - /unlink 4368 (time 1775198953.40 total 30.02 last 155.94) total: 4500 /unlink in 30.97 seconds: 145.32 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 806 (time 1775198968.91 total 10.00 last 80.56) - open/close 1670 (time 1775198978.92 total 20.01 last 86.36) - open/close 2533 (time 1775198988.93 total 30.02 last 86.21) - open/close 3338 (time 1775198998.95 total 40.04 last 80.35) - open/close 4126 (time 1775199008.95 total 50.04 last 78.80) total: 4500 open/close in 54.67 seconds: 82.32 ops/second - /unlink 1602 (time 1775199028.53 total 10.00 last 160.20) - /unlink 3039 (time 1775199038.54 total 20.00 last 143.63) - /unlink 4400 (time 1775199048.54 total 30.01 last 136.10) total: 4500 /unlink in 30.89 seconds: 145.66 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 787 (time 1775199064.62 total 10.00 last 78.70) - open/close 1640 (time 1775199074.62 total 20.00 last 85.26) - open/close 2493 (time 1775199084.63 total 30.00 last 85.30) - open/close 3430 (time 1775199094.63 total 40.01 last 93.64) - open/close 4134 (time 1775199104.64 total 50.02 last 70.34) total: 4500 open/close in 55.15 seconds: 81.59 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1710 (time 1775199124.71 total 10.01 last 170.91) - /unlink 3484 (time 1775199134.72 total 20.02 last 177.12) total: 4500 /unlink in 26.85 seconds: 167.61 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 999 (time 1775199155.39 total 10.00 last 99.89) - open/close 1924 (time 1775199165.40 total 20.01 last 92.42) - open/close 2593 (time 1775199175.41 total 30.02 last 66.82) - open/close 3155 (time 1775199185.42 total 40.03 last 56.18) - open/close 3864 (time 1775199195.43 total 50.04 last 70.83) - open/close 4401 (time 1775199205.43 total 60.04 last 53.67) total: 4500 open/close in 61.72 seconds: 72.90 ops/second - /unlink 1494 (time 1775199223.29 total 10.00 last 149.37) - /unlink 2926 (time 1775199233.30 total 20.01 last 143.12) total: 4500 /unlink in 29.28 seconds: 153.68 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 805 (time 1775199257.91 total 10.01 last 80.45) - open/close 1591 (time 1775199267.91 total 20.01 last 78.56) - open/close 2380 (time 1775199277.91 total 30.01 last 78.90) - open/close 2983 (time 1775199287.93 total 40.02 last 60.23) - open/close 3597 (time 1775199297.94 total 50.04 last 61.29) - open/close 4159 (time 1775199307.96 total 60.06 last 56.09) total: 4500 open/close in 66.12 seconds: 68.06 ops/second - /unlink 1297 (time 1775199329.83 total 10.00 last 129.67) - /unlink 2546 (time 1775199339.83 total 20.00 last 124.90) - /unlink 4065 (time 1775199349.83 total 30.01 last 151.80) total: 4500 /unlink in 32.71 seconds: 137.57 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 762 (time 1775199367.68 total 10.01 last 76.13) - open/close 1550 (time 1775199377.69 total 20.02 last 78.72) - open/close 2239 (time 1775199387.71 total 30.03 last 68.81) - open/close 2768 (time 1775199397.71 total 40.04 last 52.85) - open/close 3500 (time 1775199407.72 total 50.05 last 73.15) - open/close 4247 (time 1775199417.75 total 60.07 last 74.52) total: 4500 open/close in 63.24 seconds: 71.15 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1222 (time 1775199435.96 total 10.01 last 122.06) - /unlink 2376 (time 1775199445.97 total 20.02 last 115.30) - /unlink 3590 (time 1775199455.97 total 30.02 last 121.36) total: 4500 /unlink in 36.79 seconds: 122.31 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 725 (time 1775199477.51 total 10.01 last 72.44) - open/close 1450 (time 1775199487.51 total 20.01 last 72.49) - open/close 2308 (time 1775199497.51 total 30.01 last 85.79) - open/close 3066 (time 1775199507.52 total 40.01 last 75.77) - open/close 3965 (time 1775199517.54 total 50.04 last 89.70) total: 4500 open/close in 56.04 seconds: 80.30 ops/second - /unlink 1557 (time 1775199537.36 total 10.00 last 155.69) - /unlink 2388 (time 1775199547.37 total 20.00 last 83.07) - /unlink 3217 (time 1775199557.37 total 30.01 last 82.88) - /unlink 4112 (time 1775199567.38 total 40.02 last 89.38) total: 4500 /unlink in 42.89 seconds: 104.91 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 691 (time 1775199587.47 total 10.01 last 69.02) - open/close 1499 (time 1775199597.49 total 20.02 last 80.69) - open/close 2193 (time 1775199607.49 total 30.03 last 69.40) - open/close 2947 (time 1775199617.49 total 40.03 last 75.36) - open/close 3600 (time 1775199627.51 total 50.04 last 65.22) - open/close 4368 (time 1775199637.51 total 60.05 last 76.74) total: 4500 open/close in 61.53 seconds: 73.13 ops/second - /unlink 1191 (time 1775199654.39 total 10.00 last 119.09) - /unlink 2347 (time 1775199664.39 total 20.00 last 115.57) - /unlink 3487 (time 1775199674.39 total 30.01 last 113.96) - /unlink 4265 (time 1775199684.40 total 40.01 last 77.76) total: 4500 /unlink in 42.69 seconds: 105.41 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 531 (time 1775199704.61 total 10.00 last 53.09) - open/close 1211 (time 1775199714.63 total 20.02 last 67.86) - open/close 1875 (time 1775199724.64 total 30.03 last 66.32) - open/close 2519 (time 1775199734.66 total 40.06 last 64.26) - open/close 3274 (time 1775199744.67 total 50.06 last 75.46) - open/close 3905 (time 1775199754.68 total 60.07 last 63.06) - open/close 4453 (time 1775199764.69 total 70.08 last 54.71) total: 4500 open/close in 70.84 seconds: 63.53 ops/second - /unlink 1239 (time 1775199780.85 total 10.00 last 123.86) - /unlink 2582 (time 1775199790.85 total 20.01 last 134.25) - /unlink 4059 (time 1775199800.86 total 30.01 last 147.64) total: 4500 /unlink in 33.07 seconds: 136.07 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 546 (time 1775199820.95 total 10.01 last 54.54) - open/close 1166 (time 1775199830.96 total 20.02 last 61.92) - open/close 1944 (time 1775199840.97 total 30.03 last 77.76) - open/close 2682 (time 1775199850.99 total 40.05 last 73.66) - open/close 3372 (time 1775199860.99 total 50.05 last 68.99) - open/close 4031 (time 1775199870.99 total 60.05 last 65.89) total: 4500 open/close in 69.55 seconds: 64.70 ops/second - /unlink 1262 (time 1775199896.59 total 10.01 last 126.12) - /unlink 2539 (time 1775199906.60 total 20.01 last 127.65) - /unlink 3536 (time 1775199916.60 total 30.01 last 99.69) total: 4500 /unlink in 37.81 seconds: 119.00 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 484 (time 1775199940.42 total 10.01 last 48.33) - open/close 1101 (time 1775199950.44 total 20.03 last 61.62) - open/close 1738 (time 1775199960.45 total 30.04 last 63.62) - open/close 2398 (time 1775199970.45 total 40.04 last 65.98) - open/close 3084 (time 1775199980.46 total 50.05 last 68.54) - open/close 3677 (time 1775199990.46 total 60.05 last 59.29) - open/close 4376 (time 1775200000.47 total 70.06 last 69.86) total: 4500 open/close in 72.01 seconds: 62.49 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 680 (time 1775200018.47 total 10.01 last 67.95) - /unlink 1166 (time 1775200028.49 total 20.03 last 48.51) - /unlink 1906 (time 1775200038.51 total 30.04 last 73.89) - /unlink 2854 (time 1775200048.52 total 40.06 last 94.63) - /unlink 3679 (time 1775200058.52 total 50.06 last 82.49) total: 4500 /unlink in 59.40 seconds: 75.76 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 820 (time 1775200083.14 total 10.00 last 81.96) - open/close 1666 (time 1775200093.15 total 20.01 last 84.54) - open/close 2399 (time 1775200103.15 total 30.01 last 73.28) - open/close 3021 (time 1775200113.16 total 40.02 last 62.14) - open/close 3715 (time 1775200123.20 total 50.06 last 69.14) - open/close 4308 (time 1775200133.20 total 60.06 last 59.28) total: 4500 open/close in 63.27 seconds: 71.12 ops/second - /unlink 1635 (time 1775200151.02 total 10.00 last 163.42) - /unlink 3134 (time 1775200161.03 total 20.01 last 149.85) - /unlink 4485 (time 1775200171.04 total 30.02 last 134.97) total: 4500 /unlink in 30.19 seconds: 149.05 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 577 (time 1775200186.98 total 10.01 last 57.63) - open/close 1060 (time 1775200196.99 total 20.03 last 48.24) - open/close 1498 (time 1775200207.01 total 30.04 last 43.75) - open/close 2102 (time 1775200217.02 total 40.05 last 60.32) - open/close 2834 (time 1775200227.04 total 50.08 last 73.02) - open/close 3354 (time 1775200237.05 total 60.08 last 51.98) - open/close 4168 (time 1775200247.05 total 70.08 last 81.37) total: 4500 open/close in 74.64 seconds: 60.29 ops/second - /unlink 1196 (time 1775200266.87 total 10.00 last 119.59) - /unlink 2373 (time 1775200276.88 total 20.01 last 117.62) - /unlink 3673 (time 1775200286.88 total 30.01 last 129.95) total: 4500 /unlink in 36.72 seconds: 122.56 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11184: 94019 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg226-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg226-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg226-server stop mds service on oleg226-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg226-server stop mds service on oleg226-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg226-server unloading modules via unload_modules_local on: 'oleg226-server' oleg226-server: oleg226-server.virtnet: executing unload_modules_local oleg226-server: modules unloaded.