Archive for category yahoo finance

what’s the sitch

As I have a lot to do right am not really coming closer to my goal to write a general purpose app which can pull data from the yahoo streaming server. So I decided to post a few facts about the yahoo api. I hope this helps someone developing his app. So lets see what we have. First an explanation of these cryptic symbols:

a00: ask price
b00: bid price
g00: day’s range low
h00: day’s range high
j10: market cap
v00: volume
a50: ask size
b60: bid size
b30: ecn bid
o50: ecn bid size
z03: ecn ext hr bid
z04: ecn ext hr bid size
b20: ecn ask
o40: ecn ask size
z05: ecn ext hr ask
z07: ecn ext hr ask size
h01: ecn day’s high
g01: ecn day’s low
h02: ecn ext hr day’s high
g11: ecn ext hr day’s low
t10: last trade time, will be in unix epoch format
t50: ecnQuote/last/time
t51: ecn ext hour time
t53: RTQuote/last/time
t54: RTExthourQuote/last/time
l10: last trade
l90: ecnQuote/last/value
l91: ecn ext hour price
l84: RTQuote/last/value
l86: RTExthourQuote/last/value
c10: quote/change/absolute
c81: ecnQuote/afterHourChange/absolute
c60: ecnQuote/change/absolute
z02: ecn ext hour change
z08: ecn ext hour change
c63: RTQuote/change/absolute
c85: RTExthourQuote/afterHourChange/absolute
c64: RTExthourQuote/change/absolute
p20: quote/change/percent
c82: ecnQuote/afterHourChange/percent
p40: ecnQuote/change/percent
p41: ecn ext hour percent change
z09: ecn ext hour percent change
p43: RTQuote/change/percent
c86: RTExtHourQuote/afterHourChange/percent
p44: RTExtHourQuote/change/percent

These are the ones I found in the javascript sources. For most of it I don’t have a clue what it is for so don’t ask me what it means.

Next thing I want to mention is the result. A general approach would be to request your data and then run it through a proper JSON parser so you can work with the data as an object. Luckily this doesn’t work. The JSON returned is not 100 percent compliant with the JSON definition. Yahoo packs these symbols (seen above) not into quotes, so the parser denies to read it. A simple way around it is to string replace all possible symbols with quotes.

That’s it so far. I will post more if I have new information to share.



bringing the yahoo finance stream to the shell

A little while ago a posted a primitive way to get to yahoo finance streaming data. As you can guess this was just the beginning. To raise the bar I tried to parse the received data and bring it to the shell. To get this done I needed several tools.

  • curl – to send and receive the http request
  • transform – a primitive tool to do streaming operations within one line
  • spidermonkey shell (a javascript shell which can parse and reformat the data)

The complete logic will be done in the javascript. So lets start with the curl command line:

curl -s -o - -N ',MSFT&k=l10&callback=parent.yfs_u1f&mktmcb=parent.yfs_mktmcb&gencallback=parent.yfs_gencb'

Let’s see what we have here. First we call the yahoo streaming api and want the current price (l10) for the stocks of Sun ans Microsoft. The callback part cannot be changed. If you change this part the whole request will not succeed. Also important is to get the output to STDOUT so that we can pipe the output to the next application.

Second part of the work is just to call the transform application (further explanation here).

The third part is to pipe the output of the transform process into the javascript shell. I started the shell with the following command:

js -f script.js

The script script.js look like this:

yfs_u1f = function(tmp) {
print("msft: "+tmp.MSFT.l10);
print("java: "+tmp.JAVA.l10);

yfs_mktmcb = function(tmp) {
/*ignore timestamp */

var parent=this;
parent.yfs_u1f = yfs_u1f;
parent.yfs_mktmcb = yfs_mktmcb;

var t = readline();
if(t.substr(0,3) == "try"){

First we have to implement the callback functions which will be called from the http response. Then we construct an object called parent where we map these functions into. Now we have a working construct to receive the data and are able to work with it in our shell. What we need now is a little while loop to continuously read from STDIN and wait for new data. By the way accessing the tmp variable in the callback function seems somewhat complicated to me. I’m sure there is an easier way to access it but I have no clue how. If you have an idea how to do it better please post it to the comments.

The complete bash statement would look like this:

curl -s -o - -N ',MSFT&k=l10&callback=parent.yfs_u1f&mktmcb=parent.yfs_mktmcb&gencallback=parent.yfs_gencb' | /tmp/transform | js -f script.js

If you run this you should get this output:

msft: 17.83
java: 4.47
msft: 17.84
msft: 17.86
msft: 17.81
java: 4.46

Now you can use whatever tools you want to work with that data. For me this will be piped directly into my postgres db for further processing.

, , ,


Streaming editor improved

I recently ran into the situation that I needed a streaming editor which does not work line-wise. I was receiving an html stream and wanted to remove all the html tags.

Although I tried different existing editors like sed or replace, non of it got it right for me. All implementations waited for a newline to work with the data. I also tried disabling every possible buffer but the problem still persists.

So after quite some trial and error I started writing my own little program which could do the job. So here is the code:


int main (int argc, char* argv[])
  char c;
  int tag =0;
  do {
    c = getchar();
    if (c == '<' && tag==0) ++tag;
    if (c == '>' && tag>0) {--tag; c='\n';}
    if (tag ==0 ){
  } while (c != EOF);
  return 0;

Just compile it with the following statement and it will filter everything which comes from stdin.

gcc -fPIC -Wall -otransform transform.c

Here is a sample statement of how to use it:

curl -N -s | ./transform

, ,

1 Comment