== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 00:56:35 (1783140995) Stopping clients: oleg448-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg448-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg448-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg448-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg448-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg448-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg448-server oleg448-server: oleg448-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg448-server' oleg448-server: oleg448-server.virtnet: executing load_modules_local oleg448-server: Loading modules from /home/green/git/lustre-release/lustre oleg448-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg448-server: detected 4 online CPUs by sysfs oleg448-server: MODOPTS_LIBCFS= oleg448-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg448-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg448-server: oleg448-server.virtnet: executing set_default_debug -1 all pdsh@oleg448-client: oleg448-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg448-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg448-server: oleg448-server.virtnet: executing set_default_debug -1 all pdsh@oleg448-client: oleg448-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg448-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg448-server: oleg448-server.virtnet: executing set_default_debug -1 all pdsh@oleg448-client: oleg448-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg448-client.virtnet: -o user_xattr,flock 192.168.204.148@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush2 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 616 (time 1783141156.18 total 10.01 last 61.52) - open/close 1269 (time 1783141166.20 total 20.03 last 65.19) - open/close 1892 (time 1783141176.20 total 30.04 last 62.24) - open/close 2630 (time 1783141186.21 total 40.05 last 73.73) - open/close 3361 (time 1783141196.22 total 50.06 last 73.05) - open/close 4147 (time 1783141206.22 total 60.06 last 78.59) total: 4500 open/close in 64.55 seconds: 69.72 ops/second - /unlink 1402 (time 1783141223.95 total 10.00 last 140.14) - /unlink 2733 (time 1783141233.95 total 20.01 last 133.08) - /unlink 4209 (time 1783141243.96 total 30.02 last 147.41) total: 4500 /unlink in 31.99 seconds: 140.69 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 762 (time 1783141259.01 total 10.01 last 76.14) - open/close 1537 (time 1783141269.02 total 20.01 last 77.44) - open/close 2287 (time 1783141279.02 total 30.02 last 74.98) - open/close 3095 (time 1783141289.03 total 40.03 last 80.70) - open/close 3938 (time 1783141299.04 total 50.04 last 84.24) total: 4500 open/close in 57.14 seconds: 78.76 ops/second - /unlink 1504 (time 1783141319.22 total 10.00 last 150.34) - /unlink 3039 (time 1783141329.23 total 20.02 last 153.33) - /unlink 4486 (time 1783141339.24 total 30.02 last 144.64) total: 4500 /unlink in 30.18 seconds: 149.08 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 709 (time 1783141352.58 total 10.00 last 70.89) - open/close 1447 (time 1783141362.60 total 20.01 last 73.72) - open/close 2231 (time 1783141372.60 total 30.02 last 78.35) - open/close 3028 (time 1783141382.62 total 40.04 last 79.55) - open/close 3780 (time 1783141392.63 total 50.04 last 75.16) total: 4500 open/close in 59.26 seconds: 75.93 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1358 (time 1783141415.19 total 10.00 last 135.74) - /unlink 2698 (time 1783141425.19 total 20.01 last 133.96) - /unlink 4193 (time 1783141435.20 total 30.02 last 149.33) total: 4500 /unlink in 32.02 seconds: 140.52 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 781 (time 1783141450.43 total 10.00 last 78.08) - open/close 1584 (time 1783141460.44 total 20.01 last 80.26) - open/close 2129 (time 1783141470.46 total 30.02 last 54.41) - open/close 2685 (time 1783141480.47 total 40.04 last 55.52) - open/close 3140 (time 1783141490.48 total 50.05 last 45.46) - open/close 3425 (time 1783141500.50 total 60.07 last 28.43) - open/close 3644 (time 1783141510.54 total 70.11 last 21.81) - open/close 4019 (time 1783141520.59 total 80.16 last 37.34) - open/close 4433 (time 1783141530.63 total 90.20 last 41.22) total: 4500 open/close in 91.31 seconds: 49.28 ops/second - /unlink 1074 (time 1783141547.46 total 10.00 last 107.38) - /unlink 2246 (time 1783141557.46 total 20.01 last 117.17) - /unlink 3245 (time 1783141567.47 total 30.01 last 99.85) - /unlink 4157 (time 1783141577.47 total 40.01 last 91.18) total: 4500 /unlink in 42.81 seconds: 105.12 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 572 (time 1783141596.29 total 10.01 last 57.15) - open/close 1156 (time 1783141606.31 total 20.03 last 58.26) - open/close 1733 (time 1783141616.33 total 30.05 last 57.63) - open/close 2370 (time 1783141626.33 total 40.05 last 63.65) - open/close 2995 (time 1783141636.35 total 50.07 last 62.43) - open/close 3575 (time 1783141646.35 total 60.07 last 58.00) - open/close 4175 (time 1783141656.36 total 70.08 last 59.91) total: 4500 open/close in 75.15 seconds: 59.88 ops/second - /unlink 1180 (time 1783141676.20 total 10.00 last 117.98) - /unlink 2362 (time 1783141686.20 total 20.01 last 118.15) - /unlink 3468 (time 1783141696.20 total 30.01 last 110.57) total: 4500 /unlink in 39.36 seconds: 114.33 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 567 (time 1783141721.12 total 10.01 last 56.67) - open/close 1236 (time 1783141731.12 total 20.01 last 66.89) - open/close 1853 (time 1783141741.12 total 30.01 last 61.67) - open/close 2458 (time 1783141751.13 total 40.02 last 60.45) - open/close 3061 (time 1783141761.14 total 50.02 last 60.28) - open/close 3625 (time 1783141771.15 total 60.04 last 56.31) - open/close 4314 (time 1783141781.16 total 70.05 last 68.83) total: 4500 open/close in 73.25 seconds: 61.43 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1132 (time 1783141799.58 total 10.00 last 113.17) - /unlink 2218 (time 1783141809.58 total 20.00 last 108.58) - /unlink 3324 (time 1783141819.59 total 30.01 last 110.51) - /unlink 4421 (time 1783141829.59 total 40.01 last 109.69) total: 4500 /unlink in 41.42 seconds: 108.64 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 627 (time 1783141846.78 total 10.02 last 62.59) - open/close 1260 (time 1783141856.79 total 20.03 last 63.25) - open/close 1900 (time 1783141866.79 total 30.03 last 63.99) - open/close 2419 (time 1783141876.80 total 40.04 last 51.83) - open/close 3015 (time 1783141886.82 total 50.06 last 59.48) - open/close 3587 (time 1783141896.83 total 60.07 last 57.13) - open/close 4174 (time 1783141906.86 total 70.10 last 58.53) total: 4500 open/close in 75.18 seconds: 59.86 ops/second - /unlink 1112 (time 1783141927.14 total 10.01 last 111.12) - /unlink 2258 (time 1783141937.14 total 20.01 last 114.55) - /unlink 3238 (time 1783141947.16 total 30.03 last 97.83) - /unlink 4315 (time 1783141957.16 total 40.04 last 107.62) total: 4500 /unlink in 41.73 seconds: 107.85 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 593 (time 1783141974.57 total 10.02 last 59.19) - open/close 1256 (time 1783141984.58 total 20.03 last 66.25) - open/close 1885 (time 1783141994.58 total 30.03 last 62.88) - open/close 2518 (time 1783142004.59 total 40.04 last 63.25) - open/close 3081 (time 1783142014.59 total 50.04 last 56.29) - open/close 3690 (time 1783142024.59 total 60.04 last 60.87) - open/close 4255 (time 1783142034.61 total 70.06 last 56.43) total: 4500 open/close in 74.20 seconds: 60.64 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1282 (time 1783142053.43 total 10.01 last 128.12) - /unlink 2457 (time 1783142063.43 total 20.01 last 117.48) - /unlink 3528 (time 1783142073.43 total 30.01 last 107.07) total: 4500 /unlink in 38.67 seconds: 116.36 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 618 (time 1783142097.44 total 10.01 last 61.76) - open/close 1219 (time 1783142107.44 total 20.01 last 60.10) - open/close 1914 (time 1783142117.45 total 30.02 last 69.44) - open/close 2523 (time 1783142127.47 total 40.04 last 60.74) - open/close 3129 (time 1783142137.50 total 50.07 last 60.44) - open/close 3763 (time 1783142147.50 total 60.07 last 63.40) - open/close 4365 (time 1783142157.51 total 70.07 last 60.18) total: 4500 open/close in 72.42 seconds: 62.13 ops/second - /unlink 1099 (time 1783142174.79 total 10.02 last 109.69) - /unlink 2251 (time 1783142184.80 total 20.02 last 115.15) - /unlink 3318 (time 1783142194.80 total 30.02 last 106.69) - /unlink 4497 (time 1783142204.81 total 40.03 last 117.82) total: 4500 /unlink in 40.05 seconds: 112.36 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 652 (time 1783142219.27 total 10.00 last 65.20) - open/close 1263 (time 1783142229.27 total 20.01 last 61.06) - open/close 1895 (time 1783142239.28 total 30.01 last 63.16) - open/close 2459 (time 1783142249.28 total 40.02 last 56.39) - open/close 3073 (time 1783142259.29 total 50.02 last 61.38) - open/close 3691 (time 1783142269.29 total 60.03 last 61.74) - open/close 4315 (time 1783142279.30 total 70.04 last 62.36) total: 4500 open/close in 72.60 seconds: 61.99 ops/second - /unlink 1187 (time 1783142295.71 total 10.00 last 118.70) - /unlink 2272 (time 1783142305.71 total 20.00 last 108.47) - /unlink 3504 (time 1783142315.72 total 30.00 last 123.18) total: 4500 /unlink in 38.81 seconds: 115.96 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 618 (time 1783142339.46 total 10.01 last 61.75) - open/close 1242 (time 1783142349.46 total 20.01 last 62.40) - open/close 1899 (time 1783142359.46 total 30.01 last 65.69) - open/close 2575 (time 1783142369.46 total 40.01 last 67.59) - open/close 3242 (time 1783142379.47 total 50.02 last 66.67) - open/close 3839 (time 1783142389.47 total 60.02 last 59.69) total: 4500 open/close in 69.86 seconds: 64.41 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1265 (time 1783142413.53 total 10.00 last 126.49) - /unlink 2514 (time 1783142423.54 total 20.00 last 124.86) - /unlink 3799 (time 1783142433.54 total 30.01 last 128.46) total: 4500 /unlink in 35.44 seconds: 126.99 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 704 (time 1783142452.81 total 10.01 last 70.34) - open/close 1369 (time 1783142462.82 total 20.01 last 66.47) - open/close 2002 (time 1783142472.82 total 30.02 last 63.27) - open/close 2664 (time 1783142482.84 total 40.03 last 66.08) - open/close 3344 (time 1783142492.85 total 50.04 last 67.96) - open/close 4012 (time 1783142502.86 total 60.05 last 66.73) total: 4500 open/close in 67.00 seconds: 67.16 ops/second - /unlink 1312 (time 1783142523.54 total 10.01 last 131.11) - /unlink 2546 (time 1783142533.55 total 20.02 last 123.24) - /unlink 3807 (time 1783142543.56 total 30.03 last 126.00) total: 4500 /unlink in 36.06 seconds: 124.78 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 640 (time 1783142563.99 total 10.02 last 63.90) - open/close 1293 (time 1783142573.99 total 20.02 last 65.29) - open/close 1989 (time 1783142583.99 total 30.02 last 69.56) - open/close 2686 (time 1783142594.00 total 40.03 last 69.67) - open/close 3321 (time 1783142604.01 total 50.04 last 63.44) - open/close 3999 (time 1783142614.01 total 60.04 last 67.78) total: 4500 open/close in 67.69 seconds: 66.48 ops/second - /unlink 1178 (time 1783142636.14 total 10.00 last 117.76) - /unlink 2429 (time 1783142646.14 total 20.00 last 125.10) - /unlink 3632 (time 1783142656.15 total 30.01 last 120.20) total: 4500 /unlink in 37.07 seconds: 121.38 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 635 (time 1783142677.15 total 10.01 last 63.43) - open/close 1334 (time 1783142687.17 total 20.02 last 69.82) - open/close 2045 (time 1783142697.17 total 30.03 last 71.06) - open/close 2763 (time 1783142707.18 total 40.03 last 71.78) - open/close 3446 (time 1783142717.19 total 50.05 last 68.20) - open/close 4121 (time 1783142727.20 total 60.06 last 67.41) total: 4500 open/close in 66.01 seconds: 68.17 ops/second - /unlink 1343 (time 1783142746.97 total 10.00 last 134.24) - /unlink 2728 (time 1783142756.97 total 20.01 last 138.42) - /unlink 4106 (time 1783142766.98 total 30.02 last 137.73) total: 4500 /unlink in 32.92 seconds: 136.71 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 646 (time 1783142784.06 total 10.01 last 64.55) - open/close 1309 (time 1783142794.06 total 20.01 last 66.29) - open/close 2002 (time 1783142804.07 total 30.01 last 69.26) - open/close 2683 (time 1783142814.07 total 40.02 last 68.09) - open/close 3373 (time 1783142824.08 total 50.02 last 68.96) - open/close 4064 (time 1783142834.08 total 60.03 last 69.04) total: 4500 open/close in 66.12 seconds: 68.06 ops/second - /unlink 1264 (time 1783142854.02 total 10.00 last 126.38) - /unlink 2516 (time 1783142864.03 total 20.01 last 125.11) - /unlink 3780 (time 1783142874.03 total 30.01 last 126.39) total: 4500 /unlink in 35.85 seconds: 125.51 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 666 (time 1783142893.68 total 10.01 last 66.56) - open/close 1361 (time 1783142903.69 total 20.02 last 69.40) - open/close 2056 (time 1783142913.71 total 30.03 last 69.41) - open/close 2786 (time 1783142923.71 total 40.04 last 72.99) - open/close 3469 (time 1783142933.72 total 50.04 last 68.25) - open/close 4184 (time 1783142943.72 total 60.05 last 71.47) total: 4500 open/close in 64.50 seconds: 69.77 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1236 (time 1783142961.95 total 10.01 last 123.47) - /unlink 2526 (time 1783142971.97 total 20.03 last 128.77) - /unlink 3901 (time 1783142981.97 total 30.03 last 137.45) total: 4500 /unlink in 34.20 seconds: 131.57 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 774 (time 1783142999.52 total 10.01 last 77.32) - open/close 1580 (time 1783143009.53 total 20.02 last 80.54) - open/close 2319 (time 1783143019.53 total 30.02 last 73.87) - open/close 3038 (time 1783143029.54 total 40.03 last 71.82) - open/close 3771 (time 1783143039.55 total 50.04 last 73.23) - open/close 4498 (time 1783143049.56 total 60.05 last 72.62) total: 4500 open/close in 60.07 seconds: 74.92 ops/second - /unlink 1299 (time 1783143063.10 total 10.01 last 129.80) - /unlink 2711 (time 1783143073.10 total 20.01 last 141.12) - /unlink 4029 (time 1783143083.11 total 30.02 last 131.77) total: 4500 /unlink in 33.35 seconds: 134.95 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 792 (time 1783143100.17 total 10.01 last 79.11) - open/close 1551 (time 1783143110.18 total 20.02 last 75.82) - open/close 2309 (time 1783143120.18 total 30.03 last 75.75) - open/close 3072 (time 1783143130.20 total 40.05 last 76.17) - open/close 3866 (time 1783143140.20 total 50.05 last 79.39) total: 4500 open/close in 58.11 seconds: 77.44 ops/second - /unlink 1477 (time 1783143161.61 total 10.00 last 147.68) - /unlink 3006 (time 1783143171.61 total 20.00 last 152.89) - /unlink 4484 (time 1783143181.61 total 30.01 last 147.70) total: 4500 /unlink in 30.13 seconds: 149.35 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11242: 95061 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg448-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg448-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg448-server stop mds service on oleg448-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg448-server stop mds service on oleg448-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg448-server unloading modules via unload_modules_local on: 'oleg448-server' oleg448-server: oleg448-server.virtnet: executing unload_modules_local oleg448-server: modules unloaded.