Solaris and Linux

       $ pgrep  -lf _pmon_
       12586 ora_pmon_ee10204

ORACLE_SID is ee10204

       $ pwdx 12586
       12586: /u01/oracle/10.2.0.4/ee1/dbs

ORACLE_HOME is /u01/oracle/10.2.0.4/ee1

HPUX

       $ ps -ef | grep pmon
       ora1024 25611     1  0  Mar 21  ?         0:24 ora_pmon_itanic10

ORACLE_SID is itanic10

       $ pfiles 25611 | grep  bin
       25611:                  /opt/ora1024/app/db_1/bin/oracle

ORACLE_HOME is /opt/ora1024/app/db_1

AIX

        $ ps -ef | grep pmon
      ora1024   262314        1   0   Mar 23      -  0:12 ora_pmon_mercury

ORACLE_SID is mercury

      $ ls -l /proc/262314/cwd
      lr-x------   2 ora1024  dba  0 Mar 23 19:31 cwd -> /data/opt/app/product/10.2.0.4/db_1/dbs/

ORACLE_HOME is /data/opt/app/product/10.2.0.4/db_1

Recently the question of language efficiency verses productivity has been nagging me. The nagging question includes  what  languages, frameworks and packages are best for collecting data, analyzing data and exposing the data in a rich UI via a web browser.

Productivity

Here is an interesting graphic on speed of the programs in various languages

http://hinchcliffe.org/img/weblanguagecomparison2.png

Now the above graphic might be hard to read (wish it was larger) but what it is indicating is that Python is  roughly 10x times slower in execution time to than the equivalent programs in Java. On the other hand the expect productivity gain for a programmer is 5-10x faster in Python than Java. Now on the surface my reaction is “OK, it’s faster to program in Python, ok, but Java is 10x faster so it’s clearly the choice!”, but if I put the numbers into real life I’m like OMG – imagine a Python program that takes me 1 day to write, now that same program would take me 5-10 days in Java!? Thats a huge deal. Hands down, I’m going to program in Python (or something other than Java).  I can deal with some runtime performance issues.

Should a company program in Java or Python? What if Python would take a year to bring to market? What if the Java verison took 5-10 years ?!  When asked whether if he could go back and recreate Slideshare in some other language than Ruby, Jonathan Boutelle, said the question is mute. If they hadn’t written it in Ruby and had in Java they wouldn’t even be having the conversation. Sure Ruby had some scaling issues for them, but they are  released  Slideshare on the market and became successful.

Here is another graphic on productivity from the book “From Java to Ruby”

The data is base on a study of productivity in C, Java, Perl, Python: http://page.mi.fu-berlin.de/prechelt/Biblio//jccpprt_computer2000.pdf

As far as productivity comparisons, I think the following two images of Python vs Java explain it nicely:

http://blog.chenzhongke.com/2010/05/why-python-is-more-productive-than-java.html

For a down and out dirty view Java sprawl that further exacerbates productivity check out:

http://chaosinmotion.com/blog/?p=622

(as a rebuttal try pursing hello world in gnu packaging : http://ftp.gnu.org/gnu/hello/hello-2.6.tar.gz )

Performance

Here is a fascinating site with empirical data on speed of languages

http://shootout.alioth.debian.org/

Current speed might be a bit misleading as popularity will impact the efforts put into the issues of a language and a language that is popular though maybe less efficient will see improvements, for example it’s interesting to see the improvements in Java over the years, and now to see the improvements in Ruby.  Though Ruby is slower than  Scala and Scala’s productivity may even be better than Ruby, Scala doesn’t, yet, have the market momentum, thus the assurances that it is a language to invest in now.

Also if speed of the language is an issue the solution is not to throw the baby out with the bathwater, ie throw out the productive programming language altogether and go for a fast one such as C or JAVA but to find the areas of slowness in the productive framework and replace those with a fast function in another language, ie polyglot programming.

Popularity

another interesting image on the current usage of languages:

http://www.dataists.com/2010/12/ranking-the-popularity-of-programming-langauges/

Languages mentioned in job postings as % and % increase

http://www.soa-at-work.com/2010/02/it-job-trends-which-technologies-you.html

The issues of showing total growth  verses % growth lend itself well to heat map representation. Here is a heat map from O’Reilly:

(Interesting to all the Oracle folks: PL/SQL shows a 51% growth 2007-2008.

Here is data from O’Reilly on growth over time, showing Python and C# growing steadily:

http://radar.oreilly.com/2009/02/state-of-the-computer-book-mar-22.html

http://radar.oreilly.com/2011/02/2010-book-market-4.html

Tiobe Index

http://www.tiobe.com/index.php/content/paperinfo/tpci/index.html

Here is google trends normalized by C programming (with out normalization all trends are down because google trends show hits normalized by all traffic and as general traffic increases verses computer geek searches general computer geek searches trend downwards)

Interesting how Ruby seems to be trending down (as normalized against C  searches)

another perspective: Google Insights:

Just for Fun

here  is how I started my morning, a fun video on Python, that catalyzed the above discussion:

other links

hello world in different JVM/CLR languages http://carlosqt.blogspot.com/2010/06/most-active-net-and-jvm-languages.html

People ask me what the limit is to the number of tables  in a SQL statement  that DB Optimizer can handle when tuning.  Above is a tiny thumbnail for a 100 table SQL statemement. The SQL actually only has a select from one view  but that view expands to 100 different tables. Pretty awesome.

The first step of such a display would be selecting the data in such a way to represent the graphic. A graphic should be harder to do than a similar, though less powerful, ascii representation.

So here is a quick sketch of looking at redo generation over time. Right now I have to monitor how well redo can be archived which depends on how it fast it is generated. (typically what I’m more interested in is the general performance which is what the above graphic addressed)

The output below shows the redo generated per hour on average (picking the maximum hour in the bucket which is 4 hours long below) and the maximum generation at the minute level.  I may or may not be able to archive redo at the up to the minute but I do want to make sure I can keep up with each hour.

DAY    HOUR  MAX(MB)    w1     w2      w3       w4       w5
SAT     0       9               5       4       4       3
SAT     4       12              4       5       4       4
SAT     8       1               0       0       0       0
SAT     12      0               0       0       0       0
SAT     16      1               0       0       0       0
SAT     20      4               1       0       0       1
SUN     0       10              3       3       1       3
SUN     4       13              6       5       3       5
SUN     8       7               6       6       0       6
SUN     12      7               0       3       0       4
SUN     16      1               0       0       1       0
SUN     20      8               3       2       0       3
MON     0       8               3       2       3       2
MON     4       7               2       3       2       1
MON     8       5               1       0       2       0
MON     12      1               0       0       1       0
MON     16      1               0       0       0       0
MON     20      7               2       2       0       2
TUE     0       14              6       5       7       4
TUE     4       7               1       1       1       2
TUE     8       3               0       0       0       0
TUE     12      1       1       0       0       0       0
TUE     16      1       1       0       0       0       0
TUE     20      3       1       1       1       1       1
WED     0       8       3       2       3       2       2
WED     4       7       2       1       3       2       2
WED     8       8       1       0       0       2       3
WED     12      7       1       0       0       1       1
WED     16      1       1       0       0       0       1
WED     20      4       1       1       1       1       1
THU     0       15      7       8       6       6       6
THU     4       8       2       1       1       1       1
THU     8       1       1       0       0       0       1
THU     12      16      1       11      0       0       1
THU     16      1       1       0       0       0       1
THU     20      4       1       1       1       1       1
FRI     0       11      2       2       2       2       2
FRI     4       8       3       1       1       1       1
FRI     8       4       1       0       0       0       0

set heading off
set feedback off
set pagesize 0

with pivot_data as (
   select
          WW pivot_col
        , DY
        , D
        , HH
        , max(maxval) mv
        , max(average) av
   from
      ( select distinct
               begin_time,
               4*trunc(to_char(begin_time,'HH24')/4)     HH,
               to_char(begin_time,'DY')       DY,
               mod(to_char(begin_time,'D'),7)  D,
               mod(to_char(begin_time,'WW'),5)       WW,
               average/(1024*1024) average,
               maxval/(1024*1024) maxval,
               snap_id
        from   DBA_HIST_SYSMETRIC_SUMMARY
        where  dbid=&&DBID and
               metric_name='Redo Generated Per Sec'
	       and begin_time > sysdate - 31
       )
   group by HH,D,DY,WW
)
select DY ||'	'||
       HH ||'	'||
       round(max(mv),0)    ||'	'||
       round(max(w1),0)  ||'	'||
       round(max(w2),0)  ||'	'||
       round(max(w3),0)  ||'	'||
       round(max(w4),0)  ||'	'||
       round(max(w5),0)
from (
   select *
   from  pivot_data
   pivot ( avg(av)
           for pivot_col in ( 1 as w1,2 as w2,3 as w3 ,4 as w4 ,5 as w5 )
         )
)
group by DY,D,HH
order by D,HH
/

set heading on
set feedback on
set pagesize 30

set pagesize 100
col DY for A4
col HH for 99
col mx for 99
col w1 for 99
col w2 for 99
col w3 for 99
col w4 for 99
col w5 for 99

with pivot_data as (
   select
          WW
        , DY
        , D
        , HH
        , max(maxval) mv
        , max(average) av
   from
      ( select distinct
               begin_time,
               4*trunc(to_char(begin_time,'HH24')/4)     HH,
               to_char(begin_time,'DY')       DY,
               mod(to_char(begin_time,'D'),7)  D,
               mod(to_char(begin_time,'WW'),5)       WW,
               average/(1024*1024) average,
               maxval/(1024*1024) maxval,
               snap_id
        from   DBA_HIST_SYSMETRIC_SUMMARY
        where
               metric_name='Redo Generated Per Sec'
           and begin_time > sysdate - 31
       )
   group by HH,D,DY,WW
)
select DY, HH,
      round(max( mv ),0) mx,
      round(max( decode( WW, 0 , av, null ) ),0) w1,
      round(max( decode( WW, 1 , av, null ) ),0) w2,
      round(max( decode( WW, 2 , av, null ) ),0) w3,
      round(max( decode( WW, 3 , av, null ) ),0) w4,
      round(max( decode( WW, 4 , av, null ) ),0) w5
   from  pivot_data
group by DY,D,HH
order by D,HH
/

TM                  AAS    CPU     IO   WAIT GRAPH
---------------- ------ ------ ------ ------ --------------------------------------------------------------------------------
03-FEB  04:00:00    3.6    1.8    1.8     .1 +++++++++ooooooooo                      8
03-FEB  05:00:00    4.1    1.9    2.1     .1 ++++++++++oooooooooo-                   8
03-FEB  06:00:00    5.1    2.8    1.9     .3 ++++++++++++++oooooooooo--              8
03-FEB  07:00:00    5.2    2.6    2.2     .3 +++++++++++++ooooooooooo--              8
03-FEB  08:00:00    5.9    3.0    2.6     .2 +++++++++++++++ooooooooooooo-           8
03-FEB  09:00:00    6.3    3.2    2.8     .3 ++++++++++++++++oooooooooooooo--        8
03-FEB  10:00:00    6.3    3.0    2.9     .3 +++++++++++++++ooooooooooooooo--        8
03-FEB  11:00:00    5.5    2.9    2.5     .2 ++++++++++++++oooooooooooo-             8
03-FEB  12:00:00    5.3    2.6    2.5     .2 +++++++++++++ooooooooooooo-             8
03-FEB  13:00:00    5.4    2.7    2.4     .3 +++++++++++++oooooooooooo-              8
03-FEB  14:00:00    5.5    2.8    2.4     .3 ++++++++++++++oooooooooooo-             8
03-FEB  15:00:00    6.1    2.8    3.1     .2 ++++++++++++++ooooooooooooooo-          8
03-FEB  16:00:00    6.3    3.1    3.0     .2 +++++++++++++++ooooooooooooooo-         8
03-FEB  17:00:00    5.6    2.7    2.6     .2 ++++++++++++++ooooooooooooo-            8
03-FEB  18:00:00    3.4    2.1    1.1     .1 +++++++++++oooooo-                      8
03-FEB  19:00:00    4.7    2.5    1.6     .6 +++++++++++++oooooooo---                8
03-FEB  20:00:00    4.7    2.8    1.8     .1 ++++++++++++++ooooooooo                 8
03-FEB  21:00:00    7.3    3.6    3.3     .4 ++++++++++++++++++ooooooooooooooooo--   8
03-FEB  22:00:00   19.2    6.4   11.5    1.2 ++++++++++++++++++++++++++++++++oooooooo8ooooooooooooooooooooooooooooooooooooooo
03-FEB  23:00:00   12.7    5.2    7.1     .5 ++++++++++++++++++++++++++oooooooooooooo8oooooooooooooooooooooo--
04-FEB  00:00:00   11.1    4.5    6.3     .3 ++++++++++++++++++++++oooooooooooooooooo8oooooooooooooo--
04-FEB  01:00:00   12.8    4.3    5.9    2.6 ++++++++++++++++++++++oooooooooooooooooo8oooooooooooo-------------
04-FEB  02:00:00    4.2    2.5    1.6     .1 ++++++++++++oooooooo                    8
04-FEB  03:00:00    2.1    1.3     .7     .1 +++++++ooo                              8
04-FEB  04:00:00    2.1    1.3     .9     .0 ++++++oooo                              8
04-FEB  05:00:00    2.3    1.2    1.0     .1 ++++++ooooo                             8
04-FEB  06:00:00    3.7    2.5    1.0     .2 +++++++++++++ooooo-                     8

Def v_days=10   -- amount of days to cover in report
Def v_secs=3600 -- size of bucket in seconds, ie one row represents avg over this interval
Def v_bars=5    -- size of one AAS in characters wide
Def v_graph=80  -- width of graph in characters

undef DBID
col graph format a&v_graph
col aas format 999.9
col total format 99999
col npts format 99999
col wait for 999.9
col cpu for 999.9
col io for 999.9

/* dba_hist_active_sess_history */

select
        to_char(to_date(tday||' '||tmod*&v_secs,'YYMMDD SSSSS'),'DD-MON  HH24:MI:SS') tm,
        --samples npts,
        round(total/&v_secs,1) aas,
        round(cpu/&v_secs,1) cpu,
        round(io/&v_secs,1) io,
        round(waits/&v_secs,1) wait,
	-- substr, ie trunc, the whole graph to make sure it doesn't overflow
        substr(
	   -- substr, ie trunc, the graph below the # of CPU cores line
           -- draw the whole graph and trunc at # of cores line
	   substr(
	     rpad('+',round((cpu*&v_bars)/&v_secs),'+') ||
             rpad('o',round((io*&v_bars)/&v_secs),'o')  ||
             rpad('-',round((waits*&v_bars)/&v_secs),'-')  ||
             rpad(' ',p.value * &v_bars,' '),0,(p.value * &v_bars)) ||
        p.value  ||
	   -- draw the whole graph, then cut off the amount we drew before the # of cores
           substr(
	     rpad('+',round((cpu*&v_bars)/&v_secs),'+') ||
             rpad('o',round((io*&v_bars)/&v_secs),'o')  ||
             rpad('-',round((waits*&v_bars)/&v_secs),'-')  ||
             rpad(' ',p.value * &v_bars,' '),(p.value * &v_bars),( &v_graph-&v_bars*p.value) )
        ,0,&v_graph)
        graph
from (
   select
       to_char(sample_time,'YYMMDD')                   tday
     , trunc(to_char(sample_time,'SSSSS')/&v_secs) tmod
     , (max(sample_id) - min(sample_id) + 1 )      samples
     , sum(decode(session_state,'ON CPU',10,decode(session_type,'BACKGROUND',0,10)))  total
     , sum(decode(session_state,'ON CPU' ,10,0))    cpu
     , sum(decode(session_state,'WAITING',10,0)) -
       sum(decode(session_type,'BACKGROUND',decode(session_state,'WAITING',10,0)))    -
       sum(decode(event,'db file sequential read',10,
                        'db file scattered read',10,
                        'db file parallel read',10,
                        'direct path read',10,
                        'direct path read temp',10,
                        'direct path write',10,
                        'direct path write temp',10, 0)) waits
     , sum(decode(session_type,'FOREGROUND',
		 decode(event,'db file sequential read',10,
                                  'db file scattered read',10,
                                  'db file parallel read',10,
                                  'direct path read',10,
                                  'direct path read temp',10,
                                  'direct path write',10,
                                  'direct path write temp',10, 0))) IO
       /* for waits I want to subtract out the BACKGROUND
          but for CPU I want to count everyone */
   from
      dba_hist_active_sess_history
   where sample_time > sysdate - &v_days
   and dbid=&&DBID
   and sample_time < (select min(sample_time) from v$active_session_history)
   group by  trunc(to_char(sample_time,'SSSSS')/&v_secs),
             to_char(sample_time,'YYMMDD')
) ash,
  ( select  value from dba_hist_parameter where DBID=&&DBID and parameter_name='cpu_count' and rownum < 2 ) p
order by to_date(tday||' '||tmod*&v_secs,'YYMMDD SSSSS')
/

It’s fun to see how graphics are evolving.

The title of this post is from one of my favorite blogs Information is Beautiful along with a similar blog information aesthetics.

Above from left to right are

• My Linkedin network
• Population migration
• Race and Ethnicity map of Washington DC

I just like them for the way the display information. I’m more interested in systems performance data but I like these visualizations

The Linkedin visualization has roots in jflowmap which is used to display the second graphic. The  third graphic is from a contact on flickr who I think is doing interesting displays with geodata.

(some of the prices I got off of programmers paradise, others are word of mouth, so any clarifications on prices would be appreciated)

The tools marked “yes” are ones that I use or would use.  I would use Quest’s Performance Analyzer given the chance. I’ve never used it but from the demos of the tool, I like what I see.  I have never used Mumbai, but plan to and like again what I see in the demos.

All the other tools marked “yes” I use.

All the tools except “other stuff”, I consider reasonable tools for one job or another. Some are better than others in different ways. The “other stuff” tools I don’t see much point in.

I generally consider OEM with diagnostic pack too expensive and slow, but hey, if customers have it, I’ll use it ! (I better like it as I designed the performance page and top activity page ) Plus diag pack comes with v$active_session_history and all of AWR which I can query directly with OEM.

I tried to design “DB Optimizer” to be simple to install on my laptop and just point at databases and immediately start monitoring and tuning.  The biggest advantage of “DB Optimizer” over other tools, for me, is the  Visual SQL Tuning diagrams.  Other nice features are being able to load test code by multiple concurrent sessions.  With concurrent sessions running code examples, I can easily demonstrate issues that arise in the database with the profiling page.

But for absolute ease of install and speed, I have to hand it to Lab128. Lab128 runs fast, runs it’s own ASH collection (which doesn’t require diag pack) and has the option to read v$active_session_history as well.

Though Lab128 is fast, collects ASH info and can monitor multiple databases simultaneously for days on end, it still doesn’t allow me a real easily accessible centralized database of performance data. That’s why I created “S-ASH” which can collect ASH data from multiple databases into a central repository where I can run analytic queries or visualize it with ASHMon.

ASHMon and  S-ASH are still rough pieces of code as I spent the last 3 years working solely on DB-Optimizer.

Karl Arao
Tim Gorman

http://www.groundworkopensource.com

http://www.zenoss.com

Ganglia

10053 Viewer (replace .doc with .zip and unzip)

xplan – extend explain plan info from Alberto Dell Era

latchprof - Tanel’s latch collection query

waitprof - Tanel’s wait collection query

tvdxstat - Trivadis Extended Tracefile Analysis Tool

spviewer this looks interesting for statspack/AWR viewing and comparison

enteros does some statspack/AWR trending and other monitoring

I wonder how many smaller companies are out there with Oracle monitoring tools.

Thanks to Tanel Poder and Greg Rahn  for the template idea and for tweaks in the wordpress theme. The theme was tweaked by Greg and then repurposed by Tanel who help me get a copy and then get started.

For plugins, I used

• Blogger Importer – import blogspot posts and comments
• SBS Blogroll – for  blogspot type blogroll (I tweaked the code to be able to change the title of people’s blogs)
• amr shortcode any widget – to put blogroll on it’s own page
• SyntaxHighlighter Evolved – for marking up SQL code (as well as other code)
• W3 Total Cache – improve load times
• Akismet – anti spam
• Google Analytics for WordPress

Such accelerated learning has been outlined in the book the “Power of Pull” by John Hagel III , John Seely Brown and Lang Davison.

An individual or even a fixed team will have at in general linear output when things are going well, but when different  groups  and far flung individuals work together in tight feedback loops, output, learning and creation can accellerate.

The acceleration is happening more and more with the communication and collaboration that is enables and facilitated by the web.

This collaborative tight feedback loop is what I want to see happen in the database performance tuning arena  and graphical monitoring and what I think I see the beginnings of in  Marcus Mönnig’s Mumbai tool. Marcus creates his own Oracle database tuning tool but also pulls in the work of Tanel Poder’s Snapper package and Egor Starostin Orasrp.

Dan Rosensweig former COO at Yahoo responding to the question: Why did Google succeed in search over Yahoo?

We were many things to many people.
They were one great thing to everybody
Sometimes it's just what you believe in.
It's how the question get's phrased.
When I came to yahoo people were not interested
in investing in search because there was no way
to make money.
...
Google didn't care about making money
They cared about improving peoples ability to search
and they were originally going to be enterprise.
Yahoo was actually using their search engine.
Then something happened.
Somebody created a business model around search.
To get around patents Google improved on Overtures business model.
The logic [Yahoo] had was you bid the most you come up first.
It was sort of self policing.
Why would you bid the most to come up first
if you weren't the most relevant?
Google figured something out.
It's not who bids the most,
its how you generate the most amount of value
off each click.
Its a combination of location and price and click through.
That enabled them to make more money on every click
than we did which allowed them to  invest a lot more money in increased
monetization engines and then buy distribution.
They could buy distribution at half the cost that we could
or a third the cost that we could.
Here we were as a public company trying to balance multiple
businesses and they were focused on one
and they were making more money on every click
and they were able to run the table successfully in search.

Why do companies succeed? Big question with lots of answers.
Google and the mass media seem to propagate the story that Google’s search is better and that’s why Google won. Clean and simple. Black and white. People like black and white, but the story is more complicated than that. The search space was competitive when Google came on the scene. I personally used Altavista before Google came one the scene and well after. I compared many of the search results and I was always impressed with Altavista. Google finally won my heart when the came out with maps.google.com. The simple map interface was so logical and so simple that it put mapquest to shame for good reason.

But the story of search much more complicated and interesting. For more reading check out

http://www.searchenginehistory.com/