OMNeT++/OMNEST Bug Tracker

View Issue Details Jump to Notes ] Issue History ] Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0000012OMNeT++IDE / result analysispublic2008-11-28 14:572009-09-09 09:25
Reporterbaumgart 
Assigned Toandras 
PrioritynormalSeverityfeatureReproducibilityN/A
StatusfeedbackResolutionreopened 
PlatformOSOS Version
Product Version4.0b8 
Target VersionFixed in Version4.0rc2 
Summary0000012: Easier generation of scalar plots
DescriptionI'm wondering if the beautiful new results analysis tool could be used to generate fast plots out of the scenario description in the ini file.

Take for example the following ini file with an additional [Plot] section:

[General]
repeat = 2
**.lifetimeMean = ${lifetime=1000s,10000s}

[Config ChordChurn]
**.overlayType = "ChordModules"
**.stabilizeInterval = ${stab=5s,20s}

[Config KademliaChurn]
**.overlayType = "KademliaModules"

[Plot ChurnComparison]
x = lifetime
y = "*Sent Total Bytes*"
lines = KademliaChurn, ChordChurn-stab=5s, ChordChurn-stab=20s

The idea is, that this could generate a plot with the input parameter "lifetime" on the x axes and the scalar matching the (unambiguous) string "*Sent Total Bytes*" on the y axes. The plot should contain three lines taking the results from the scenarios KademliaChurn and ChordChurn. Because the scenario ChordChurn has a second input parameter "stab", it has to be specified, too.

The lines should represent the averages of all repetitions. The title and axes labels could be also generated from the plot ("ChurnComparison", "lifetime", "Sent Total Bytes").

Maybe this can already be accomplished with the GUI of the results analysis tool, but probably is a lot more complicated than the ini file approach (at least I haven't found an easy way to do this with the GUI).
TagsNo tags attached.
Attached Filespatch file icon scatterplot_scave.patch [^] (12,904 bytes) 2009-03-24 18:37 [Show Content]

- Relationships

-  Notes
(0000039)
rhornig (administrator)
2008-12-01 18:45

This example is a very simple charting case. Once the you start to create more complex charts with processing, scatter plot etc. a simple syntax like this will not be enough to describe the scenario. This case and a lot more can be created with the IDE.

Please check the resultfiles sample project. Open PureAlohaExperiment and open the DataSets page. Double click on Scatter chart Load vs. Utilization. This diagram is similar to the one you have described.
(0000047)
baumgart (reporter)
2008-12-02 18:23

I've tried again to create a plot for my simple example with the IDE, but failed: I can't select the ${lifetime} parameter as "X data" in the "Create ScatterChart Object" dialog. After looking at the resultfiles sample project, I've noticed the following line in the sca file:
"scalar . mean 1"

But in my .sca file there is no "scalar . lifetime 1000" (although there is an "attr lifetime 1000s", which seems to be ignored for scatter plots).

Maybe you could add a short step-by-step tutorial to the wiki how to make a scatter plot similar to my example.

BTW, for our kind of simulations the simple syntax covers almost any scalar plotting case we can image. What kind of more complex examples do you think of, that are supported by the GUI, which are not covered by the simple ini syntax?
(Please keep I'm talking about scalars, not vector data).
(0000049)
andras (administrator)
2008-12-03 14:37

We looked into it. To get the "lifetime and "stab" iteration variables recorded in the .sca file as scalar, the "s" has to be written outside the ${} in the ini file:

[General]
repeat = 2
**.lifetimeMean = ${lifetime=1000,10000}s

Currently only entirely numeric values get recorded (because scalars are dimensionless), i.e. 1000 is OK but 1000s is not... This is a bit subtle, we'll look into how we can improve it.
(0000050)
andras (administrator)
2008-12-03 15:38

OK now I implemented that iteration variables that look like numbers with units (e.g. "100s") will also be recorded, as a scalar that has a "unit" attribute (this attr was already supported for normal scalars as well).

scalar . lifetime 1000
attr unit s

This is somewhat unsafe though, since the Analysis Tool in the IDE currently ignores scalar attributes; so nothing stops you from shooting yourself in the foot by writing "${lifetime=1s,10s,30s,1min,2min}" (which will appear on the scatter plot as 1, 10, 30, 1, 2).
(0000053)
baumgart (reporter)
2008-12-03 16:42

Thanks, now I was able generate a very simple plot (KademliaChurn in the example above). But I'm still struggling how to add the other two lines to the (same) plot (ChordChurn in the example above). Probably I have to create 3 different datasets? Do I have to filter on stab=5s and stab=20s to have these as two separate lines?
(0000065)
baumgart (reporter)
2008-12-05 18:12

Okay, after spending some time reading the code of the scavetool if succeded in filtering all runs with stab=5s with the following filter:

attr:measurement("$stab=5, $liftime=*") and name("*Sent Total Bytes*")

But I've still haven't found a solution to plot these three lines in a single graph. Any clues?

And how to i export the values of the graph to csv?
(0000072)
andras (administrator)
2008-12-07 10:52

In the past days we fixed two related problems:
1. inifile editor: referring to "inherited" iteration variables was reported as an error, while it is in fact not
2. incorrect legend for scalar plots.
Also created a brief example for scalar plots, I'll attach it soon.
(0000083)
baumgart (reporter)
2008-12-09 14:07

We're still looking for an easy solution to produce a simple scalars plot like given in the example above. Since the IDE currently doesn't support the generation of publication-quality plots (e.g. eps or pdf output), we probably have to take gnuplot or matplotlib for plot generation.

Since most result analysis features of the IDE seem to simply call scavetool methods, we're trying to use the scavetool library to export datafiles, which can be used with gnuplot. Unfortunately the library is quite complex with little documentation yet, so maybe you could give us some hints how to use it:

We mainly need these features:
- Read all scalar files
- calculate the average for repetitions (optional output as sca file would be great, e.g. ChordChurn-*.sca -> ChordChurn.sca)
- group by experiment name and measurement (ChordChurn-stab=5)
- create a two-dimensional array with the remaining measurement parameter (lifetime in this example) as different rows and all scalars as different columns
- export as csv

I think most of these features are already present in scavetool, but I don't get the big picture how to connect these. How do I average all repetitions? How do I generate the array? Maybe prepareScatterPlot2() seems to do this...

I don't know how hard it will be to built a solution based on scavetool, but currently I don't think it makes sense to port our old omnetpp3.x python scripts to omnetpp4.x.
(0000090)
andras (administrator)
2008-12-11 00:01

> We're still looking for an easy solution to produce a simple scalars plot like given in the example above.

Put this under [Config KademliaChurn]:

description = ${stab=0}

And in the analysis tool, check $stab as "Iso line" variable in the scalar plot's properties dialog. (The above inifile line is a kludge to get $stab listed in the "Iso lines" listbox -- it only lists itervars that are present in every run.)

> Since the IDE currently doesn't support the generation of publication-quality plots (e.g. eps or pdf output)

This will be done in 4.1.

> We probably have to take gnuplot or matplotlib for plot generation.

We are planning to have some interfacing with R. Besides statistics functions it is also a good programming language, and can generate publication-quality plots.
(0000097)
baumgart (reporter)
2008-12-16 14:46

Okay, this trick somehow helped. I still don't see any experiment names in the chart, but probably the one with stab=0 is the KademliaChurn experiment. And I don't know if this approach is feasible with multiple experiments and many different itervars.

So what is probably missing is a "Iso line" setting for experiment names, isn't it?

I still don't what to filter in the scatterPlot tool with "Iso lines" and what to filter in the dataset view - and I have a bad feeling that maybe several experiments get grouped to the same iso line. I also don't completely understand this mixture using itervars parameters as scalar output values.

So what we really want is to average repetitions of the same experiment (maybe also with min/max value and deviation to generate errorbars) and then sort this to plot one itervar on the x axis and one output scalar on the y axis.

The most important thing currently missing is csv export of result files, so we can generate plots with gnuplot.
(0000099)
rhornig (administrator)
2008-12-20 18:01

You are right. Maybe an iso-line-name=whatever name
would be usable for the scalar plots. Additionally the chartes should be modified to show also this value (iso-line-name) in the dropdown box.

For the moment you are right that you can only identify the lines with the different stab values (and also right that this would cause problems with several conigs with very different parameters.

A possible solution:
[Config one]
iso-line-name= Line name one

[Config two]
iso-line-name= Line name two with par: ${paramvalue}

As for CSV export: right click on any (may select more) lines in the "Browse data" view or at any pint in the dataset tree and the corresponding dataset can be exported in CSV, Matlab or Octave format
(0000100)
baumgart (reporter)
2008-12-20 21:46

I don't think that you have to extend the ini or sca format, but only the way scavetool and the IDE produce scatter plots. It should be sufficient if the scavetool would natively use experiment names and itervars (which are already present in all sca files) as iso lines for scatter plots.

I know that you can export dataset values. But we miss support to export averaged and sorted scatter plot values.
(0000101)
baumgart (reporter)
2008-12-22 23:24

I've found another issue with using iso line settings to generate separate lines:
It only works with numeric itervars and not with strings.
(0000112)
andras (administrator)
2009-01-19 16:06

Run attributes can also be selected for iso lines in the dialog. This makes it possible do the plots in the bug description.
(0000135)
baumgart (reporter)
2009-03-24 18:37

I've attached a patch to generate scatter plot output for plotting with gnuplot.

You have to specify a single itervar for the x axis - all remaining itervars and different experiments are sorted to different columns.

Replications of the same measurement are averaged and a 95% or 99% confidence interval is calculated.

The first line of the output contains labels for plotting.
(0000170)
baumgart (reporter)
2009-09-09 09:25

OverSim now contains several Python scripts which can be used to generate scalar and vector plots with gnuplot (with support for confidence intervals and convex hulls for scatterplots).

Since this solution is faster and much cleaner than the proposed patch to scavetool, this patch should not be used anymore. Bug should be closed...

- Issue History
Date Modified Username Field Change
2008-11-28 14:57 baumgart New Issue
2008-12-01 18:45 rhornig Note Added: 0000039
2008-12-01 18:45 rhornig Status new => closed
2008-12-01 18:45 rhornig Resolution open => won't fix
2008-12-01 18:45 rhornig Fixed in Version => 4.0rc1
2008-12-02 18:23 baumgart Note Added: 0000047
2008-12-02 18:23 baumgart Status closed => feedback
2008-12-02 18:23 baumgart Resolution won't fix => reopened
2008-12-03 14:37 andras Note Added: 0000049
2008-12-03 15:38 andras Note Added: 0000050
2008-12-03 16:42 baumgart Note Added: 0000053
2008-12-05 18:12 baumgart Note Added: 0000065
2008-12-07 10:52 andras Note Added: 0000072
2008-12-09 14:07 baumgart Note Added: 0000083
2008-12-11 00:01 andras Note Added: 0000090
2008-12-16 14:46 baumgart Note Added: 0000097
2008-12-20 18:01 rhornig Note Added: 0000099
2008-12-20 18:02 rhornig Status feedback => postponed
2008-12-20 18:02 rhornig Resolution reopened => suspended
2008-12-20 18:02 rhornig Fixed in Version 4.0rc1 => 4.1
2008-12-20 21:46 baumgart Note Added: 0000100
2008-12-20 21:46 baumgart Status postponed => feedback
2008-12-20 21:46 baumgart Resolution suspended => reopened
2008-12-22 23:24 baumgart Note Added: 0000101
2009-01-19 16:06 andras Note Added: 0000112
2009-01-19 16:06 andras Status feedback => resolved
2009-01-19 16:06 andras Fixed in Version 4.1 => 4.0rc2
2009-01-19 16:06 andras Resolution reopened => fixed
2009-01-19 16:06 andras Assigned To => andras
2009-03-24 18:37 baumgart Note Added: 0000135
2009-03-24 18:37 baumgart Status resolved => feedback
2009-03-24 18:37 baumgart Resolution fixed => reopened
2009-03-24 18:37 baumgart File Added: scatterplot_scave.patch
2009-09-09 09:25 baumgart Note Added: 0000170


Copyright © 2000 - 2019 MantisBT Team
Powered by Mantis Bugtracker