Difference between pages "r7.1.1:Usage Guide (benchmarking)" and "r7.1.1:Usage Guide (libbenchmark)"

From liblfds.org
(Difference between pages)
Jump to navigation Jump to search
m (1 revision imported)
 
m (1 revision imported)
 
Line 1: Line 1:
{{DISPLAYTITLE:Usage Guide (benchmarking)}}
{{DISPLAYTITLE:Usage Guide (libbenchmark)}}
==Introducton==
==Introduction==
Benchmarking functionality is implemented in a library, ''libbenchmark'', which offers a small and simple API, so it can be used on arbitrary platforms.  Using the ''libbenchmark'' library also requires using one API exposed by the ''libshared'' library, for memory management.  Finally, a convenience thin command line veneer, ''benchmark'', is provided, which is used from the command line to run the test suite and see the results.
This page describes how to use the ''libbenchmark'' library.


As such, usage is either the use of the ''libbenchmark'' API (and also the ''libshared'' API, for memory management), or the use of the command line ''benchmark'' programme.
The library implements a great deal of functionality, almost all of which is used and only used by the ''libbenchmark'' library itself.  From the point of view of an external caller to its API, there are only a few functions; a couple to init and run the entire benchmark suite, a few more to handle the results from a benchmark run and one or two miscellanous functions.


===API Usage Guides===
==Usage==
* [[r7.1.1:Usage Guide (libshared)|Usage Guide (libshared)]]
To use ''libbenchmark'', include the header file ''libbenchmark.h'' and link as normal to the library in your build.
* [[r7.1.1:Usage Guide (libbenchmark)|Usage Guide (libbenchmark)]]


===Command Line Usage Guide===
==Dependencies==
* [[r7.1.1:Usage Guide (benchmark)|Usage Guide (benchmark)]]
The ''libbenchmark'' libraries depends on the ''libshared'' library and the ''liblfds711'' library.
 
==Source Files==
└── test_and_benchmark
    └── libbenchmark
        ├── inc
        │  └── libbenchmark
        │      ├── libbenchmark_benchmarksuite.h
        │      ├── libbenchmark_enums.h
        │      ├── libbenchmark_gnuplot.h
        │      ├── libbenchmark_results.h
        │      ├── libbenchmark_topology.h
        │      └── libbenchmark_topology_node.h
        └── src
            ├── libbenchmark_benchmarksuite
            │  ├── libbenchmark_benchmarksuite_cleanup.c
            │  ├── libbenchmark_benchmarksuite_gnuplot.c
            │  ├── libbenchmark_benchmarksuite_init.c
            │  ├── libbenchmark_benchmarksuite_internal.h
            │  └── libbenchmark_benchmarksuite_run.c
            ├── libbenchmark_results
            │  ├── libbenchmark_results_cleanup.c
            │  ├── libbenchmark_results_get_result.c
            │  ├── libbenchmark_results_init.c
            │  └── libbenchmark_results_internal.h
            ├── libbenchmark_topology
            │  ├── libbenchmark_topology_cleanup.c
            │  ├── libbenchmark_topology_init.c
            │  ├── libbenchmark_topology_internal.h
            │  ├── libbenchmark_topology_numa.c
            │  └── libbenchmark_topology_lpsets.c
            └── libbenchmark_topology_node
                ├── libbenchmark_topology_node_cleanup.c
                └── libbenchmark_topology_node_init.c
 
This is a small subset of the full set of files, and shows only those files used by the publically exposed APIs.
 
==Defines==
#define [[r7.1.1:define LIBBENCHMARK_BENCHMARKSUITE_OPTION_DURATION|LIBBENCHMARK_BENCHMARKSUITE_OPTION_DURATION]]
 
==Enums==
enum [[r7.1.1:enum libbenchmark_benchmark_id|libbenchmark_benchmark_id]];
enum [[r7.1.1:enum libbenchmark_datastructure_id|libbenchmark_datastructure_id]];
enum [[r7.1.1:enum libbenchmark_lock_id|libbenchmark_lock_id]];
enum [[r7.1.1:enum libbenchmark_topology_node_type|libbenchmark_topology_node_type]];
enum [[r7.1.1:enum libbenchmark_topology_numa_mode|libbenchmark_topology_numa_mode]];
 
==Opaque Structures==
struct [[r7.1.1:struct lfds711_list_aso_element|lfds711_list_aso_element]];
struct [[r7.1.1:struct lfds711_list_aso_state|lfds711_list_aso_state]];
struct [[r7.1.1:struct lfds711_list_asu_state|lfds711_list_asu_state]];
struct [[r7.1.1:struct libbenchmark_gnuplot_options|libbenchmark_gnuplot_options]];
struct [[r7.1.1:struct libbenchmark_results_state|libbenchmark_results_state]];
struct [[r7.1.1:struct libbenchmark_benchmarksuite_state|libbenchmark_benchmarksuite_state]];
struct [[r7.1.1:struct libbenchmark_topology_state|libbenchmark_topology_state]];
struct [[r7.1.1:struct libbenchmark_topology_node_state|libbenchmark_topology_node_state]];
struct [[r7.1.1:struct libshared_memory_state|libshared_memory_state]];
 
==Macros==
#define [[r7.1.1:macro LIBBENCHMARK_TOPOLOGY_NODE_SET_TYPE|LIBBENCHMARK_TOPOLOGY_NODE_SET_TYPE]]( tns, node_type )
#define [[r7.1.1:macro LIBBENCHMARK_TOPOLOGY_NODE_SET_LOGICAL_PROCESSOR_NUMBER|LIBBENCHMARK_TOPOLOGY_NODE_SET_LOGICAL_PROCESSOR_NUMBER]]( tns, processor_number )
#define [[r7.1.1:macro LIBBENCHMARK_TOPOLOGY_NODE_SET_WINDOWS_GROUP_NUMBER|LIBBENCHMARK_TOPOLOGY_NODE_SET_WINDOWS_GROUP_NUMBER]]( tns, win_group_number )
#define [[r7.1.1:macro LIBBENCHMARK_TOPOLOGY_NODE_UNSET_WINDOWS_GROUP_NUMBER|LIBBENCHMARK_TOPOLOGY_NODE_UNSET_WINDOWS_GROUP_NUMBER]]( tns )
#define [[r7.1.1:macro LIBBENCHMARK_GNUPLOT_OPTIONS_INIT|LIBBENCHMARK_GNUPLOT_OPTIONS_INIT]]( gpo )
#define [[r7.1.1:macro LIBBENCHMARK_GNUPLOT_OPTIONS_SET_Y_AXIS_SCALE_TYPE_LOGARITHMIC|LIBBENCHMARK_GNUPLOT_OPTIONS_SET_Y_AXIS_SCALE_TYPE_LOGARITHMIC]]( gpo )
#define [[r7.1.1:macro LIBBENCHMARK_GNUPLOT_OPTIONS_SET_WIDTH_IN_PIXELS|LIBBENCHMARK_GNUPLOT_OPTIONS_SET_WIDTH_IN_PIXELS]]( gpo, wip )
#define [[r7.1.1:macro LIBBENCHMARK_GNUPLOT_OPTIONS_SET_HEIGHT_IN_PIXELS|LIBBENCHMARK_GNUPLOT_OPTIONS_SET_HEIGHT_IN_PIXELS]]( gpo, wip )
 
==Prototypes==
int [[r7.1.1:function libbenchmark_topology_init|libbenchmark_topology_init]]( struct libbenchmark_topology_state *ts,
                                struct libshared_memory_state *ms );
void [[r7.1.1:function libbenchmark_topology_cleanup|libbenchmark_topology_cleanup]]( struct libbenchmark_topology_state *ts );
void [[r7.1.1:function libbenchmark_topology_generate_deduplicated_logical_processor_sets|libbenchmark_topology_generate_deduplicated_logical_processor_sets]]( struct libbenchmark_topology_state *ts,
                                                                          struct libshared_memory_state *ms,
                                                                          struct lfds711_list_asu_state *lp_sets );
void [[r7.1.1:function libbenchmark_topology_generate_numa_modes_list|libbenchmark_topology_generate_numa_modes_list]]( struct libbenchmark_topology_state *ts,
                                                      enum libbenchmark_topology_numa_mode numa_mode,
                                                      struct libshared_memory_state *ms,
                                                      struct lfds711_list_asu_state *numa_modes_list );
void [[r7.1.1:function libbenchmark_topology_node_init|libbenchmark_topology_node_init]]( struct libbenchmark_topology_node_state *tns );
void [[r7.1.1:function libbenchmark_topology_node_cleanup|libbenchmark_topology_node_cleanup]]( struct libbenchmark_topology_node_state *tns,
                                          void (*element_cleanup_callback)(struct lfds711_list_aso_state *lasos,
                                                                          struct lfds711_list_aso_element *lasoe) );
void [[r7.1.1:function libbenchmark_results_init|libbenchmark_results_init]]( struct libbenchmark_results_state *rs,
                                struct libshared_memory_state *ms );
void [[r7.1.1:function libbenchmark_results_cleanup|libbenchmark_results_cleanup]]( struct libbenchmark_results_state *rs );
int [[r7.1.1:function libbenchmark_results_get_result|libbenchmark_results_get_result]]( struct libbenchmark_results_state *rs,
                                      enum libbenchmark_datastructure_id datastructure_id,
                                      enum libbenchmark_benchmark_id benchmark_id,
                                      enum libbenchmark_lock_id lock_id,
                                      enum libbenchmark_topology_numa_mode numa_mode,
                                      struct lfds711_list_aso_state *lpset,
                                      struct libbenchmark_topology_node_state *tns,
                                      lfds711_pal_uint_t *result );
void [[r7.1.1:function libbenchmark_benchmarksuite_init|libbenchmark_benchmarksuite_init]]( struct libbenchmark_benchmarksuite_state *bss,
                                        struct libbenchmark_topology_state *ts,
                                        struct libshared_memory_state *ms,
                                        enum libbenchmark_topology_numa_mode numa_mode,
                                        lfds711_pal_uint_t options_bitmask,
                                        lfds711_pal_uint_t benchmark_duration_in_seconds );
void [[r7.1.1:function libbenchmark_benchmarksuite_cleanup|libbenchmark_benchmarksuite_cleanup]]( struct libbenchmark_benchmarksuite_state *bss );
void [[r7.1.1:function libbenchmark_benchmarksuite_run|libbenchmark_benchmarksuite_run]]( struct libbenchmark_benchmarksuite_state *bss,
                                      struct libbenchmark_results_state *rs );
void [[r7.1.1:function libbenchmark_benchmarksuite_get_list_of_gnuplot_strings|libbenchmark_benchmarksuite_get_list_of_gnuplot_strings]]( struct libbenchmark_benchmarksuite_state *bss,
                                                              struct libbenchmark_results_state *rs,
                                                              char *gnuplot_system_string,
                                                              struct libbenchmark_gnuplot_options *gpo,
                                                              struct lfds711_list_asu_state *list_of_gnuplot_strings );
 
==Overview==
It has become apparent from creating this page that the API interface to benchmarking is too complicated.  The API to run the benchmarks and get hold of the results is actually simple enough - two init functions (one for the benchmark and one for the results) and then a function to run the benchmarks.  The problem come when offering APIs to query the results.
 
There are many data structures, each of which can have many benchmarks, where each benchmark is run both for the ''liblfds'' lock-free version and then for the full range of system provided locking mechanisms, where each run of the benchmark for a given lock type runs over many combinations of logical cores, where for every combination a result is stored for each logical processor; and all of this on a NUMA system with more than one NUMA node is run twice, once NUMA aware and once not, to show how much difference this makes to performance.
 
As such, values for all of these parameters must be specified to the function which queries the result state - and this exposes a great deal of complexity and functionality, as can be seen above.
 
It will take a day to document this library, and the effort to do so is futile, because it will still be too complex to use.  As such, the effort will go instead into simplifying the API.  However, this should not delay the release of 7.1.1, so the functions in this library will for now go undocumented.


==See Also==
==See Also==
* [[r7.1.1:Release_7.1.1_Documentation|Release 7.1.1 Documentation]]
* [[r7.1.1:Usage Guide (benchmarking)|Usage Guide (benchmarking)]]

Latest revision as of 20:16, 17 February 2017

Introduction

This page describes how to use the libbenchmark library.

The library implements a great deal of functionality, almost all of which is used and only used by the libbenchmark library itself. From the point of view of an external caller to its API, there are only a few functions; a couple to init and run the entire benchmark suite, a few more to handle the results from a benchmark run and one or two miscellanous functions.

Usage

To use libbenchmark, include the header file libbenchmark.h and link as normal to the library in your build.

Dependencies

The libbenchmark libraries depends on the libshared library and the liblfds711 library.

Source Files

└── test_and_benchmark
    └── libbenchmark
        ├── inc
        │   └── libbenchmark
        │       ├── libbenchmark_benchmarksuite.h
        │       ├── libbenchmark_enums.h
        │       ├── libbenchmark_gnuplot.h
        │       ├── libbenchmark_results.h
        │       ├── libbenchmark_topology.h
        │       └── libbenchmark_topology_node.h
        └── src
            ├── libbenchmark_benchmarksuite
            │   ├── libbenchmark_benchmarksuite_cleanup.c
            │   ├── libbenchmark_benchmarksuite_gnuplot.c
            │   ├── libbenchmark_benchmarksuite_init.c
            │   ├── libbenchmark_benchmarksuite_internal.h
            │   └── libbenchmark_benchmarksuite_run.c
            ├── libbenchmark_results
            │   ├── libbenchmark_results_cleanup.c
            │   ├── libbenchmark_results_get_result.c
            │   ├── libbenchmark_results_init.c
            │   └── libbenchmark_results_internal.h
            ├── libbenchmark_topology
            │   ├── libbenchmark_topology_cleanup.c
            │   ├── libbenchmark_topology_init.c
            │   ├── libbenchmark_topology_internal.h
            │   ├── libbenchmark_topology_numa.c
            │   └── libbenchmark_topology_lpsets.c
            └── libbenchmark_topology_node
                ├── libbenchmark_topology_node_cleanup.c
                └── libbenchmark_topology_node_init.c

This is a small subset of the full set of files, and shows only those files used by the publically exposed APIs.

Defines

#define LIBBENCHMARK_BENCHMARKSUITE_OPTION_DURATION

Enums

enum libbenchmark_benchmark_id;
enum libbenchmark_datastructure_id;
enum libbenchmark_lock_id;
enum libbenchmark_topology_node_type;
enum libbenchmark_topology_numa_mode;

Opaque Structures

struct lfds711_list_aso_element;
struct lfds711_list_aso_state;
struct lfds711_list_asu_state;
struct libbenchmark_gnuplot_options;
struct libbenchmark_results_state;
struct libbenchmark_benchmarksuite_state;
struct libbenchmark_topology_state;
struct libbenchmark_topology_node_state;
struct libshared_memory_state;

Macros

#define LIBBENCHMARK_TOPOLOGY_NODE_SET_TYPE( tns, node_type )
#define LIBBENCHMARK_TOPOLOGY_NODE_SET_LOGICAL_PROCESSOR_NUMBER( tns, processor_number )
#define LIBBENCHMARK_TOPOLOGY_NODE_SET_WINDOWS_GROUP_NUMBER( tns, win_group_number )
#define LIBBENCHMARK_TOPOLOGY_NODE_UNSET_WINDOWS_GROUP_NUMBER( tns )

#define LIBBENCHMARK_GNUPLOT_OPTIONS_INIT( gpo )
#define LIBBENCHMARK_GNUPLOT_OPTIONS_SET_Y_AXIS_SCALE_TYPE_LOGARITHMIC( gpo )
#define LIBBENCHMARK_GNUPLOT_OPTIONS_SET_WIDTH_IN_PIXELS( gpo, wip )
#define LIBBENCHMARK_GNUPLOT_OPTIONS_SET_HEIGHT_IN_PIXELS( gpo, wip )

Prototypes

int libbenchmark_topology_init( struct libbenchmark_topology_state *ts,
                                struct libshared_memory_state *ms );
void libbenchmark_topology_cleanup( struct libbenchmark_topology_state *ts );

void libbenchmark_topology_generate_deduplicated_logical_processor_sets( struct libbenchmark_topology_state *ts,
                                                                         struct libshared_memory_state *ms,
                                                                         struct lfds711_list_asu_state *lp_sets );
void libbenchmark_topology_generate_numa_modes_list( struct libbenchmark_topology_state *ts,
                                                     enum libbenchmark_topology_numa_mode numa_mode,
                                                     struct libshared_memory_state *ms,
                                                     struct lfds711_list_asu_state *numa_modes_list );

void libbenchmark_topology_node_init( struct libbenchmark_topology_node_state *tns );
void libbenchmark_topology_node_cleanup( struct libbenchmark_topology_node_state *tns,
                                         void (*element_cleanup_callback)(struct lfds711_list_aso_state *lasos,
                                                                          struct lfds711_list_aso_element *lasoe) );

void libbenchmark_results_init( struct libbenchmark_results_state *rs,
                                struct libshared_memory_state *ms );
void libbenchmark_results_cleanup( struct libbenchmark_results_state *rs );
int libbenchmark_results_get_result( struct libbenchmark_results_state *rs,
                                     enum libbenchmark_datastructure_id datastructure_id,
                                     enum libbenchmark_benchmark_id benchmark_id,
                                     enum libbenchmark_lock_id lock_id,
                                     enum libbenchmark_topology_numa_mode numa_mode,
                                     struct lfds711_list_aso_state *lpset,
                                     struct libbenchmark_topology_node_state *tns,
                                     lfds711_pal_uint_t *result );

void libbenchmark_benchmarksuite_init( struct libbenchmark_benchmarksuite_state *bss,
                                       struct libbenchmark_topology_state *ts,
                                       struct libshared_memory_state *ms,
                                       enum libbenchmark_topology_numa_mode numa_mode,
                                       lfds711_pal_uint_t options_bitmask,
                                       lfds711_pal_uint_t benchmark_duration_in_seconds );
void libbenchmark_benchmarksuite_cleanup( struct libbenchmark_benchmarksuite_state *bss );
void libbenchmark_benchmarksuite_run( struct libbenchmark_benchmarksuite_state *bss,
                                      struct libbenchmark_results_state *rs );
void libbenchmark_benchmarksuite_get_list_of_gnuplot_strings( struct libbenchmark_benchmarksuite_state *bss,
                                                              struct libbenchmark_results_state *rs,
                                                              char *gnuplot_system_string,
                                                              struct libbenchmark_gnuplot_options *gpo,
                                                              struct lfds711_list_asu_state *list_of_gnuplot_strings );

Overview

It has become apparent from creating this page that the API interface to benchmarking is too complicated. The API to run the benchmarks and get hold of the results is actually simple enough - two init functions (one for the benchmark and one for the results) and then a function to run the benchmarks. The problem come when offering APIs to query the results.

There are many data structures, each of which can have many benchmarks, where each benchmark is run both for the liblfds lock-free version and then for the full range of system provided locking mechanisms, where each run of the benchmark for a given lock type runs over many combinations of logical cores, where for every combination a result is stored for each logical processor; and all of this on a NUMA system with more than one NUMA node is run twice, once NUMA aware and once not, to show how much difference this makes to performance.

As such, values for all of these parameters must be specified to the function which queries the result state - and this exposes a great deal of complexity and functionality, as can be seen above.

It will take a day to document this library, and the effort to do so is futile, because it will still be too complex to use. As such, the effort will go instead into simplifying the API. However, this should not delay the release of 7.1.1, so the functions in this library will for now go undocumented.

See Also