Nicer formating at https://sites.google.com/site/oraclemonitor/r-slicing-and-dicing-data

1. R data types
2. Converting columns into vectors
3. Extracting Rows and converting Rows to numeric vectors
4. Entering data
5. Vectorwise maximum/minimum
6. Column Sums and Row Sums

R can do some awesome data visualizations: http://gallery.r-enthusiasts.com/thumbs.php

Instead of doing one off data visualizations like with Excel, R can automate the process allowing one to visualize many sets of data with the same visualizations.

Installing R is pretty easy http://scs.math.yorku.ca/index.php/R:_Getting_started_with_R

There are lots of blogs out there on getting started with R. The one thing that I didn’t find explained well was slicing and dicing data.

Lets take some data that I want to visualize.  The following data shows the performance of network throughput. The throughput is measured by latency of communication in milliseconds (avg_ms) and throughput in MB per second (MB/s).

The parameters are the I/O message size in KB (0KB is actually 1 byte) and the number of concurrent threads sending data (threads)

If this data is a file, it can be easily loaded and charted with R.

Find out what directory R is working in:

go to a directory with my data and R files:

list files

load data into a variable

Simple, et voila, the data is loaded. To see the data just type the name of the variable ( the “>” is the R prompt, like “SQL>” in SQL*Plus)

Creating a chart is a breeze, just say plot(x,y) where x and y are the values you want to plot.
How to we extract an x and y from mydata?
First pick what to plot. Let’s plot averge ms latency (avg_ms) verse MB per sec (MB.s).
Here is how to extract those columns from the data

Now plot

huh … what’s that Error?

If we look at x and/or y, they are actually columns from mydata and plot() wants rows (actually vectors but we’ll get there).

To transpose a column into a row we can use “t()”

Now we can try plotting again:

and voila

but let’s address the issue of transforming x and y from columns to rows and specifically into vectors.
Let’s look at the original data and then the transformed data

Look at the datatypes of x and t(x) using the class() function

the column is considered a “data.frame” and the row is considered a “matrix”.

The method of extracting a column by it’s column name only works for datatype class data.frame.

If the datatype was a matrix we would be required to supply both the row and column as in  matrix[“row”,”column”]

By leaving either row or column empty but keeping the comma in place then it acts as a wild card.

matrix[,”column”] – gives all values in that column

matrix[“row”,] – gives all the values in that row

plot() wants a vector (but it forgivingly works with rows of data as we did above).

R data types

What are these datatypes in R?
There is a simple discussion of data types at http://www.statmethods.net/input/datatypes.html

The types are basically (using “value1:value2” gives a list iterating from value1 to value2 by increments of 1)

• integer
  • > i=1:5
> class(i)
[1] "integer"
> i
[1] 1 2 3 4 5
• character
  • > c=letters[1:5]
> class(c)
[1] "character"
> c
[1] "a" "b" "c" "d" "e"
• (booleans are integers )
  • > b=FALSE:TRUE
> class(b)
[1] "integer"
> b
[1] 0 1
• vectors
  • > v=c(1,2,3,4,5)
> class(v)
[1] "numeric"
> v
[1] 1 2 3 4 5
• matrix
  • > m=matrix(c(1,2,3,4,5))
> class(m)
[1] "matrix"
> m
     [,1]
[1,]    1
[2,]    2
[3,]    3
[4,]    4
[5,]    5
• data.frames – mixes numeric and character
  • > df=matrix(1:5,letters[1:5])      # matrix can't contain character and numeric
Error in matrix(1:5, letters[1:5]) : non-numeric matrix extent
>
> df=data.frame(1:5,letters[1:5])  # dataframe can
> class(df)
[1] "data.frame"
> df
  X1.5 letters.1.5.
1    1            a
2    2            b
3    3            c
4    4            d
5    5            e
• lists – like an matrix but can mix different data types together such as character, number, matrix

  •  > a = c(1,2,5.3,6,-2,4) # numeric vector
    > # generates 5 x 4 numeric matrix 
    > y=matrix(1:20, nrow=5,ncol=4)
    > # example of a list with 4 components - 
    > # a string, a numeric vector, a matrix, and a scaler 
    > w= list(name="Fred", mynumbers=a, mymatrix=y, age=5.3)
    > w
    $name
    [1] "Fred"
    
    $mynumbers
    [1]  1.0  2.0  5.3  6.0 -2.0  4.0
    
    $mymatrix
         [,1] [,2] [,3] [,4]
    [1,]    1    6   11   16
    [2,]    2    7   12   17
    [3,]    3    8   13   18
    [4,]    4    9   14   19
    [5,]    5   10   15   20
    
    $age
    [1] 5.3

  • extract the various parts of a list with  list[[“name”]], as in w[[“mymatrix”]]
• array – are matrices with more than 2 dimensions
• factors

Useful functions on data types

• dput(var) – will give structure of var
• class(var) – will tell the data type
• dim(var) – will set dimension
• as.matrix(data.frame) – useful for changing a data.frame into a matrix, though be careful because if there are any character values in the data frame then all entries in the matrix will be charcter

Sometimes R transforms data in ways I don’t predict, but the best strategy is just to force R to do what I want more explicitly.

Converting columns into vectors

When originally selecting out the columns of the data, we could have selected out vectors directly instead of selecting a column and transforming the column to a vector.
Instead of asking for the column which gives a column we can ask for every value in that column
by adding in a “,” infront of the column name. The brackets take the equivalent of x and y coordinates or row and column position. By adding a “,” with no value before it, we are giving a wild card to the row identifier and saying give me all the values for all rows in the column “avg_ms”

We can also extract the values by the column position instead of column name. The “avg_ms” is column 3

A third way to get the vector format is using “[[ ]]” syntax

A forth way is with the matrix$col syntax

Another way that we’ll talk about in converting a row to a vector is the apply() and as.numeric() functions:
The function apply can also change a column to a vector

These vector extractions work for columns but things are different for rows.

Extracting Rows and converting Rows to numeric vectors

The other side other coin is extracting a row into vector format. In mydata, the rows don’t have names, so we have to use position. By specifying row position with no following column names then all column values are given for that row.

The resulting data is a  data frame and not a vector  (ie a vector is of datatype numeric)
We can use the “as.numeric” function to convert the data.frame to a vector, ie numeric.
The apply() function will apply the “as.numeric” function to multiple values at once. The apply() takes 3 args

• input variable
• 1=row,2=col,1:2=both
• function to apply

see http://nsaunders.wordpress.com/2010/08/20/a-brief-introduction-to-apply-in-r/

The above applies  the change to all columns in the given row in a data.frame.

(apply can also be used for example to change all 0 to NULLs

new_matrix = apply(matrix,1:2,function(x)if (x==0)  NULL else x)

see http://stackoverflow.com/questions/3505701/r-grouping-functions-sapply-vs-lapply-vs-apply-vs-tapply-vs-by-vs-aggrega)

For selecting the row out directly as a vector, the as.matrix() function can also be used

yet another way

( see http://stackoverflow.com/questions/2545228/converting-a-dataframe-to-a-vector-by-rows)

or yet

Filtering Data

The data in the CSV file actually represents throughput not only at different I/O send sizes but also for different number of concurrent senders. What if I wanted to just plot the throughput by I/O send size for tests with one thread? How would I filter the data?

 

How about plotting the throughput by I/O size for each number of threads test?
The parameter ‘type=”o”‘  makes the plot a line plot

 

 

 

Entering data

Instead of entering data via a CSV file it can be entered directly into R

The bizarre thing about this is that the nrows corresponds to the number of columns and the matrix comes out transposed. Using t() can re-transpose it, but this is all confusing.
To make it more intuitive add the argument
"byrow=TRUE,"
and add a
"NULL"
for the rowname position in the row and columns name section

Vectorwise maximum/minimum

 

Column Sums and Row Sums

to sum up rows or colums use “rowSums()” and   “colSUms()”

http://stat.ethz.ch/R-manual/R-patched/library/base/html/colSums.html

For more info

for more info on data types and manipulation see

see: http://cran.r-project.org/doc/manuals/R-intro.html

Kyle Hailey
Uncategorized