== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 03:12:43 (1787123563) Stopping clients: oleg207-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg207-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg207-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg207-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg207-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg207-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg207-server oleg207-server: oleg207-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg207-server' oleg207-server: oleg207-server.virtnet: executing load_modules_local oleg207-server: Loading modules from /home/green/git/lustre-release/lustre oleg207-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg207-server: detected 4 online CPUs by sysfs oleg207-server: MODOPTS_LIBCFS= oleg207-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg207-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg207-server: oleg207-server.virtnet: executing set_default_debug -1 all pdsh@oleg207-client: oleg207-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg207-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg207-server: oleg207-server.virtnet: executing set_default_debug -1 all pdsh@oleg207-client: oleg207-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg207-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg207-server: oleg207-server.virtnet: executing set_default_debug -1 all pdsh@oleg207-client: oleg207-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg207-client.virtnet: -o user_xattr,flock 192.168.202.107@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H fnv_1a_64 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 796 (time 1787123782.88 total 10.00 last 79.56) - open/close 1632 (time 1787123792.88 total 20.01 last 83.59) - open/close 2441 (time 1787123802.89 total 30.02 last 80.78) - open/close 3116 (time 1787123812.90 total 40.03 last 67.45) - open/close 3778 (time 1787123822.91 total 50.03 last 66.16) - open/close 4429 (time 1787123832.91 total 60.04 last 65.06) total: 4500 open/close in 61.47 seconds: 73.20 ops/second - /unlink 1397 (time 1787123851.17 total 10.00 last 139.70) - /unlink 2998 (time 1787123861.17 total 20.00 last 160.04) - /unlink 4427 (time 1787123871.18 total 30.01 last 142.87) total: 4500 /unlink in 30.71 seconds: 146.52 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 876 (time 1787123886.67 total 10.01 last 87.53) - open/close 1801 (time 1787123896.69 total 20.03 last 92.31) - open/close 2735 (time 1787123906.70 total 30.04 last 93.33) - open/close 3534 (time 1787123916.71 total 40.04 last 79.86) - open/close 4366 (time 1787123926.72 total 50.05 last 83.10) total: 4500 open/close in 51.52 seconds: 87.34 ops/second - /unlink 1437 (time 1787123941.88 total 10.00 last 143.68) - /unlink 3050 (time 1787123951.88 total 20.00 last 161.29) - /unlink 4191 (time 1787123961.89 total 30.01 last 114.01) total: 4500 /unlink in 32.28 seconds: 139.40 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 856 (time 1787123978.70 total 10.00 last 85.60) - open/close 1522 (time 1787123988.71 total 20.01 last 66.52) - open/close 2226 (time 1787123998.72 total 30.02 last 70.34) - open/close 3011 (time 1787124008.73 total 40.03 last 78.46) - open/close 3653 (time 1787124018.74 total 50.04 last 64.13) - open/close 4471 (time 1787124028.74 total 60.04 last 81.73) total: 4500 open/close in 60.30 seconds: 74.62 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1350 (time 1787124043.06 total 10.00 last 134.94) - /unlink 2642 (time 1787124053.07 total 20.01 last 129.14) - /unlink 3853 (time 1787124063.07 total 30.01 last 121.05) total: 4500 /unlink in 34.87 seconds: 129.04 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 716 (time 1787124081.96 total 10.02 last 71.47) - open/close 1540 (time 1787124091.96 total 20.02 last 82.39) - open/close 2246 (time 1787124101.96 total 30.02 last 70.57) - open/close 3005 (time 1787124111.97 total 40.03 last 75.85) - open/close 3836 (time 1787124121.98 total 50.03 last 83.06) total: 4500 open/close in 59.65 seconds: 75.44 ops/second - /unlink 1533 (time 1787124146.96 total 10.00 last 153.28) - /unlink 3004 (time 1787124156.96 total 20.00 last 147.08) - /unlink 4482 (time 1787124166.96 total 30.00 last 147.79) total: 4500 /unlink in 30.09 seconds: 149.55 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 832 (time 1787124181.32 total 10.01 last 83.12) - open/close 1707 (time 1787124191.32 total 20.02 last 87.44) - open/close 2528 (time 1787124201.32 total 30.02 last 82.09) - open/close 3226 (time 1787124211.33 total 40.02 last 69.78) - open/close 3979 (time 1787124221.33 total 50.03 last 75.25) total: 4500 open/close in 58.07 seconds: 77.50 ops/second - /unlink 1460 (time 1787124243.74 total 10.00 last 145.98) - /unlink 2911 (time 1787124253.74 total 20.00 last 145.08) - /unlink 4299 (time 1787124263.74 total 30.01 last 138.73) total: 4500 /unlink in 31.58 seconds: 142.51 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 639 (time 1787124280.62 total 10.01 last 63.82) - open/close 1384 (time 1787124290.62 total 20.02 last 74.47) - open/close 2135 (time 1787124300.63 total 30.02 last 75.05) - open/close 2994 (time 1787124310.63 total 40.03 last 85.86) - open/close 3894 (time 1787124320.64 total 50.04 last 89.93) total: 4500 open/close in 56.51 seconds: 79.64 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1247 (time 1787124341.85 total 10.00 last 124.69) - /unlink 2723 (time 1787124351.85 total 20.00 last 147.60) - /unlink 4175 (time 1787124361.85 total 30.01 last 145.13) total: 4500 /unlink in 32.57 seconds: 138.18 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 758 (time 1787124379.16 total 10.01 last 75.70) - open/close 1553 (time 1787124389.16 total 20.02 last 79.47) - open/close 2270 (time 1787124399.16 total 30.02 last 71.70) - open/close 2891 (time 1787124409.18 total 40.03 last 61.99) - open/close 3533 (time 1787124419.21 total 50.07 last 64.00) - open/close 4482 (time 1787124429.21 total 60.07 last 94.87) total: 4500 open/close in 60.23 seconds: 74.71 ops/second - /unlink 1420 (time 1787124444.68 total 10.00 last 142.00) - /unlink 2822 (time 1787124454.68 total 20.00 last 140.20) - /unlink 4316 (time 1787124464.68 total 30.00 last 149.39) total: 4500 /unlink in 31.77 seconds: 141.63 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 715 (time 1787124481.50 total 10.00 last 71.47) - open/close 1530 (time 1787124491.50 total 20.00 last 81.50) - open/close 2305 (time 1787124501.51 total 30.01 last 77.48) - open/close 3021 (time 1787124511.51 total 40.01 last 71.60) - open/close 3760 (time 1787124521.52 total 50.02 last 73.82) total: 4500 open/close in 59.55 seconds: 75.57 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1390 (time 1787124546.56 total 10.01 last 138.91) - /unlink 2659 (time 1787124556.57 total 20.02 last 126.73) - /unlink 4104 (time 1787124566.58 total 30.02 last 144.47) total: 4500 /unlink in 33.76 seconds: 133.28 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 722 (time 1787124585.67 total 10.00 last 72.19) - open/close 1525 (time 1787124595.67 total 20.01 last 80.25) - open/close 2301 (time 1787124605.67 total 30.01 last 77.59) - open/close 3094 (time 1787124615.68 total 40.01 last 79.26) - open/close 3838 (time 1787124625.68 total 50.02 last 74.38) total: 4500 open/close in 59.22 seconds: 75.99 ops/second - /unlink 1317 (time 1787124650.02 total 10.00 last 131.67) - /unlink 2510 (time 1787124660.02 total 20.01 last 119.24) - /unlink 3844 (time 1787124670.03 total 30.01 last 133.30) total: 4500 /unlink in 37.23 seconds: 120.88 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 660 (time 1787124693.21 total 10.01 last 65.97) - open/close 1462 (time 1787124703.21 total 20.01 last 80.17) - open/close 2197 (time 1787124713.21 total 30.01 last 73.49) - open/close 3156 (time 1787124723.21 total 40.01 last 95.89) - open/close 3901 (time 1787124733.22 total 50.02 last 74.43) total: 4500 open/close in 57.82 seconds: 77.83 ops/second - /unlink 1575 (time 1787124756.04 total 10.00 last 157.49) - /unlink 2941 (time 1787124766.04 total 20.00 last 136.57) - /unlink 4234 (time 1787124776.05 total 30.01 last 129.17) total: 4500 /unlink in 31.70 seconds: 141.94 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 622 (time 1787124793.28 total 10.00 last 62.18) - open/close 1200 (time 1787124803.29 total 20.01 last 57.77) - open/close 1911 (time 1787124813.29 total 30.01 last 71.10) - open/close 2780 (time 1787124823.30 total 40.03 last 86.75) - open/close 3407 (time 1787124833.31 total 50.03 last 62.66) - open/close 4001 (time 1787124843.32 total 60.04 last 59.35) total: 4500 open/close in 67.75 seconds: 66.42 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1143 (time 1787124866.31 total 10.01 last 114.14) - /unlink 2506 (time 1787124876.32 total 20.02 last 136.24) - /unlink 4193 (time 1787124886.32 total 30.02 last 168.60) total: 4500 /unlink in 32.39 seconds: 138.94 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 629 (time 1787124905.02 total 10.01 last 62.84) - open/close 1421 (time 1787124915.05 total 20.04 last 78.93) - open/close 2108 (time 1787124925.07 total 30.06 last 68.58) - open/close 2803 (time 1787124935.10 total 40.09 last 69.28) - open/close 3633 (time 1787124945.10 total 50.09 last 82.99) total: 4500 open/close in 59.92 seconds: 75.10 ops/second - /unlink 1364 (time 1787124969.71 total 10.00 last 136.35) - /unlink 2960 (time 1787124979.72 total 20.01 last 159.52) total: 4500 /unlink in 29.75 seconds: 151.26 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 1013 (time 1787125003.74 total 10.00 last 101.28) - open/close 2060 (time 1787125013.74 total 20.00 last 104.69) - open/close 3029 (time 1787125023.77 total 30.03 last 96.61) - open/close 3885 (time 1787125033.78 total 40.04 last 85.52) total: 4500 open/close in 47.02 seconds: 95.71 ops/second - /unlink 1659 (time 1787125054.96 total 10.01 last 165.74) - /unlink 3209 (time 1787125064.97 total 20.02 last 154.82) - /unlink 4321 (time 1787125074.98 total 30.02 last 111.18) total: 4500 /unlink in 31.69 seconds: 141.99 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 756 (time 1787125092.52 total 10.00 last 75.58) - open/close 1451 (time 1787125102.55 total 20.02 last 69.34) - open/close 2133 (time 1787125112.55 total 30.03 last 68.14) - open/close 2982 (time 1787125122.56 total 40.04 last 84.83) - open/close 3748 (time 1787125132.56 total 50.04 last 76.60) total: 4500 open/close in 59.81 seconds: 75.24 ops/second - /unlink 1209 (time 1787125157.77 total 10.00 last 120.87) - /unlink 2452 (time 1787125167.77 total 20.00 last 124.28) - /unlink 3834 (time 1787125177.77 total 30.00 last 138.18) total: 4500 /unlink in 33.81 seconds: 133.10 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 870 (time 1787125196.55 total 10.01 last 86.94) - open/close 1504 (time 1787125206.56 total 20.02 last 63.33) - open/close 2137 (time 1787125216.58 total 30.03 last 63.21) - open/close 2953 (time 1787125226.58 total 40.03 last 81.59) - open/close 3710 (time 1787125236.58 total 50.04 last 75.67) - open/close 4494 (time 1787125246.59 total 60.05 last 78.32) total: 4500 open/close in 60.10 seconds: 74.88 ops/second - /unlink 1169 (time 1787125261.52 total 10.02 last 116.70) - /unlink 2607 (time 1787125271.52 total 20.02 last 143.75) - /unlink 3873 (time 1787125281.52 total 30.02 last 126.58) total: 4500 /unlink in 33.78 seconds: 133.21 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 655 (time 1787125299.80 total 10.00 last 65.49) - open/close 1427 (time 1787125309.80 total 20.01 last 77.17) - open/close 2183 (time 1787125319.81 total 30.02 last 75.53) - open/close 2865 (time 1787125329.83 total 40.03 last 68.11) - open/close 3696 (time 1787125339.84 total 50.04 last 82.97) total: 4500 open/close in 59.57 seconds: 75.55 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1254 (time 1787125363.97 total 10.00 last 125.36) - /unlink 2611 (time 1787125373.98 total 20.02 last 135.50) - /unlink 3597 (time 1787125383.98 total 30.02 last 98.59) total: 4500 /unlink in 39.05 seconds: 115.25 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 816 (time 1787125408.24 total 10.03 last 81.38) - open/close 1500 (time 1787125418.25 total 20.04 last 68.34) - open/close 2483 (time 1787125428.25 total 30.04 last 98.26) - open/close 3471 (time 1787125438.25 total 40.04 last 98.78) - open/close 4412 (time 1787125448.25 total 50.04 last 94.09) total: 4500 open/close in 50.79 seconds: 88.61 ops/second - /unlink 1582 (time 1787125463.59 total 10.00 last 158.18) - /unlink 3180 (time 1787125473.60 total 20.01 last 159.68) total: 4500 /unlink in 27.07 seconds: 166.26 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 874 (time 1787125495.56 total 10.00 last 87.36) - open/close 1720 (time 1787125505.57 total 20.01 last 84.59) - open/close 2451 (time 1787125515.57 total 30.01 last 73.06) - open/close 3283 (time 1787125525.57 total 40.01 last 83.19) - open/close 4095 (time 1787125535.59 total 50.03 last 81.04) total: 4500 open/close in 55.10 seconds: 81.67 ops/second - /unlink 1404 (time 1787125555.69 total 10.00 last 140.37) - /unlink 2813 (time 1787125565.71 total 20.01 last 140.75) - /unlink 4173 (time 1787125575.71 total 30.01 last 135.99) total: 4500 /unlink in 32.24 seconds: 139.56 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11316: 95281 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg207-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg207-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg207-server stop mds service on oleg207-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg207-server stop mds service on oleg207-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg207-server unloading modules via unload_modules_local on: 'oleg207-server' oleg207-server: oleg207-server.virtnet: executing unload_modules_local oleg207-server: modules unloaded.