Market Profile Chart using Python!

Code, ML, Python
Continuing from the earlier code  where I created the profile data using 'Alphabets, here let me try to plot the chart. This one turned out bit tricky and used an example snippet from Matplotlib for stacked bar chart!Logic is simple, instead of Alphabets, have an a number for each occurrence, use Matplotlib stacked bar chart to plot the colors.mi=defaultdict(list) TGroups=df.groupby([pd.Grouper(key='DateTime', freq=frequency)]) #iterate over each group and add to dictionary, #dictonary keys are 'High' and 'Low' of each group, #rounded and values are char A, B incremented for each period(freq group) #default 30 min since we grouped based on freq, #for each group increment the char i.e +1 min_price=np.round(df.min().Low) max_price=np.round(df.max().High) for t,g in TGroups: g_min_price=np.round(g.Low.min()) g_max_price=np.round(g.High.max()) for price in range(int(min_price), int(max_price+1)): if (price > g_max_price) or (price < g_min_price): mi[price].append(0) else:…
Read More

Market Profile using Python!

Code, ML, Python
If anyone did a technical analysis using popular trading charts, would have noticed Market Profile as one of the techniques. This was a really different way of looking at the time series data.The logic is slice the data for a period, say every '30 min' and assign an alphabet, starting with 'A' to all the 'Price'. Price is also rounded of so that we don't have too many keys based on Price. Simply put in programming parlance create a dictionary with Price(rounded of) as keys and add new Alphabet for every slice.Say first 30 min, the low is 100 and high is 110, then starting with 100 to 110, append 'A' as a value. so our dictionary would be[{100, 'A'},{101, 'A'},{102, 'A'},{103, 'A'} ..{110,'A'}],then, next  slice of 30 min, the…
Read More

All about Logistic Regression

ML
Logistic is  a classification algorithm based on probability, instead of predicting the class, it gives the log likelihood of being the given class i.e (p(y/x)  = p/(1-p))Let's look at a little bit of theory about it. We use the same linear equation only difference is, the y is binary, that is  either 0 or 1.[h_{theta}(x) =  theta_{0} + theta_{1}x]However we calculate the y using the sigmoid function to return a probability value between 0 and 1. [P(class=1) = frac{1} {1 + e^{-z}}] The reason this function returns 0 or 1 for any large positive or negative and the z would be[h_{theta}(x) = 1 / (1 + e^{-z}), z = theta_{0} + theta_{1}x]Cost function[J(theta) = sum_{i=1}^{n}y_{i}log(h_{theta}(x_{i})) + (1-y_{i})log(1-h_{theta}(x_{i}))]Here is a complete derivation, a nice one, lecture notes on logistic regressionUpdate rule Same as…
Read More

It all starts with y=mx+b

ML
  It all starts with the equation of a straight line y=mx+B. The simplest of data fitting is, using the least squares, that is minimizing the sum of squared errors(or difference between the actual and predicted). The Least Squares is in fact a partial derivative of  (f(a,b) = a + bx )  [Ref: mathworld.wolfram.com]     [frac{delta(R^2)}{ delta a} = 2 sum_{i=1}^{n}[y_i-(a+bx_i) ]= 0]  [frac{delta(R^2)}{ delta b} = 2 sum_{i=1}^{n}[y_i-(a+bx_i) ]x_i = 0]  skipping to the final equation   [a = frac {hat{y}(sum_{i=1}^{n} x_i^2) - hat{x}sum_{i=1}^{n} x_i y_i} {sum_{i=1}^{n}x_i^2 - n hat{x}^2}] [b = frac {(sum_{i=1}^{n} x_i y_i) - n hat{x} hat{y}} {sum_{i=1}^{n}x_i^2 - n hat{x}^2}] Optimization techniques use slope to minimize the error or also called cost functions in Machine learning, A cost function is something you want to minimize. In…
Read More

Campaign Efficiency – Retail Bank

ML
A leading Retail Bank wanted to know the target customer profile, so, that the marketing campaign would be effective. The target is to disburse personal load to existing and new customers. Various key data were gathered. Using AI/ML, developed a model that would predict the probability(likelihood) that a customer would take a loan, given a set of customer profile data. Data Data collected during campaign can be used to Analyze and improve the efficiency. Key variables are Age, Job, A/C balance, Previous Loan, Marital status Analyses the data and clean the raw data Compare key variables with one another to establish any correlation Establish key variables that impact the outcome using feature engineering Build a ML model using Python using key variables Train using the key variables Test it against…
Read More

HyperLedger Iroha – Setup and Configuration

Blockchain
  It wasn't straight forward for me to setup Iroha, so, I decided to document here the steps , in case I forget ( I do within a week..) HyperLedger Iroha in docker environment using a Linux system (Debian or Ubuntu): Make sure docker and docker-compose are installed in the system. If not, sudo apt-get install docker docker-compose Clone the git repository to some local directory as follows: git clone https://github.com/hyperledger/iroha -b develop In the cloned repository, from the BASH shell, run the scripts/run-iroha-dev.sh. This will perform the following (will take some time to download the docker images from network): Docker instance for the build environment with a BASH shell mounted on /opt/iroha Docker instance for the redis server with server listening in the port number 6379 Docker instance for…
Read More

Weka – Include / Exclude attributes

ML, Weka
In Weka, there may be times when one wants to remove attributes from the data, through code. While I googled and did not find any, I set upon writing my own, Here is the snippet. For example the attributes are there in a file (attributes.txt), each line is an attribute, we read each attribute and find the corresponding index in the data(ARFF buffer bbcsport.arff). We then use Remove() class setAttributeIndicesArray() to set the array of indices to be either included or excluded by setting true or false in method remove.setInvertSelection(false). BufferedReader datafile = new BufferedReader(new FileReader("bbcsport.arff")); BufferedReader attrfile = new BufferedReader(new FileReader("attributes.txt")); Instances data = new Instances(datafile); List<Integer> myList = new ArrayList<Integer>(); String line; while ((line = attrfile.readLine()) != null) { for (n = 0; n < data.numAttributes(); n++) { if…
Read More

Working with Raspberry Pi, 3g/ppp0, default gw, docker, conpot, some useful setups, etc

IoT, ML, SCADA
This is more of a bookmark for me, as I faced many setup related requirements that may be useful , the information is all over the web and one may not know it good they are. Hence this page in case I forget and am  lazy! First! Power up! Nothing happens without SD card!! Tried to install without display/keyboard/mouse by only using ssh, that's very tall requiment and ended up buying keyboard and mouse, used TV for display. Downlaod and install noobs or debian (no otherway !!) Now boot your Raspberry Pi 3 and ssh into it with user pi@x.x.x.x and the password raspberry Setup 3G USB modem, after a few hickups, got it going!  summarized here sudo apt-get update (important first step since also Wifi is setup..) lsusb and…
Read More

SCADA to IoT

IoT, ML, SCADA
Recently one of my friends called up and asked was it possible to provide a remote management and realtime monitoring of solar inverters. He added that the inverter supported ModBus. I quickly grasped this opportunity, as I saw, I can put my recently learned IoT stack to test! I spent a couple of hours to gather all the necessary info. Here is the architecture that I intend to use of the PoC.H/WRaspberry PI based MODBUS gatewayConnect inverters to the gateway through TCP/IP (RJ45)Connect Gateway to web app through Wifi or 3G (USB dongle)Switch/Router (in case of single inverter) we can try crossover cable but not advised )SoftwareImplement MODBUS communication protocol to gather data on Pi deviceRaspberry comes with 16GB local storage , use it store 48 hrs of backup dataDeploy simple dashboard for…
Read More

Who am I?

About, Ideas, Products, Startups
My colleague called and said that the company was listing software projects that were green (eco friendly) asked whether my team had done any. I remembered one such project called Green Print. This is a cool desktop application worth raving about, so, I thought why not do so? If I can't brag myself who else would!! Here it is! I quickly checked, Green Print ,to my surprise, the app is still around. It was way back 2005-06 I got a call from sales guys that there was a sales lead on printing app, I was leading (and still am) a team on MS Windows device drivers and wanted to discuss with the client. The requirement was, just a one line and simple, "develop an application to detect waste pages that are sent to the…
Read More