My docker learnings

Recently I started working on a project which includes both NodeJS and Python application. We planned to Deodorization build process for our project. During Deodorization I faced multiple issues which I eventually solved. I thought of writing a post on my docker experience to help others. Below are the some of my observations/issues I found during my journey

1. Using multiple FROM in docker file: Never ever use two FROM instructions in the Dockerfile. Docker will take only last FROM instruction. Use one FROM instruction and install another framework. In my case I use FROM for NodeJS and installed python as shown below.

WRONG WAY:

CORRECT WAY:

2. Never copy all files at the same time: This is really funny, When you install dependencies after copying all the files, dependencies will install every time when you run docker build. 

INCORRECT WAY:

Instead of copying all files, copy only dependency file (package.json for NodeJS or requirements.txt for python), install dependencies and then copy all source files.

CORRECT WAY:

3. Using RUN stead of CMD: Never use RUN command in Dockerfile, Dockefile is for creating a docker image and not executing any commands. Instead of RUN, use CMD.

4. Showing docker containers (running docker images): Most of the docker commands are similar to basic unix commands. We can use unix `ps` command to display all running docker images.

To display docker images: `docker images`

To display docker containers: `docker ps`

5. Use docker exec to run commands on docker container: Using docker exec command, we can run all basic unix commands like `ls`, `cd` etc on docker container while running the docker image. This will be very useful for checking logs in the container. 

6. Connect to the docker container: Get the current docker containers by running docker ps command. After that use exec command to connect to docker container shell prompt.

`docker exec -it CONTAINERID /bin/bash`

7. How to access host application from docker container: I spent more time compare with other topics in docker to know this. When running docker in the host and trying to access some endpoints like database from host, docker doesn't know which is host and which is docker. Need to specify the host specifically. If you want to access endpoints from host, you need to specify `host.docker.internal` instead of localhost or 127.0.0.1. This differentiates docker localhost and actual localhost.

8. Expose Port numbers: When containerizing any API end points, we need to explicitly expose port numbers in the Dockerfile. Otherwise, These ports are not accessible from host machine.

Run docker image: Run docker image by specifying  exposed ports in the run command as below.

Happy Learning!!!!

Most frequently used methods of Pandas data frame in Python!!!

Recently I worked on a machine learning project Face Recognition. Basically this project will take one image as input and return list of images which contains similar faces from input image. For this I used python framework pandas DataFrame for preprocessing images to identify good face in the image. Below are some of the data frame operations I used.

1. Create a empty DataFrame
2. Create a DataFrame from a list
3. Length of the DataFrame
4. Sorting data frame basing on specific column
5. Extract a value from DataFrame
6. Iterate through DataFrame
7. Filter/Retrieve DataFrame basing on column value
8. Extract some columns from DataFrame
9. Reading a column values (Unique) as list from DataFrame
10. Unique count from a column in DataFrame
11. Adding index column to the DataFrame
12. Delete a column from DataFrame
13. Remove duplicates from DataFrame
14. Save DataFrame as a csv file
15. Read csv file as DataFrame
16. Merge two DataFrames

17. Updating Dataframe column

18. Splitting DataFrame column

Check GitHub link for Jupyter notebook  code for above operations.

Happy Coding!!
Happy Learning!!

Useful commands for data cleaning in Machine Learning(ML) training - part 2

This post details post of all the commands specified in this post.

grep: If you are regular CLI user, you might have already knew it about grep command and its advantages. You can refer this and this for more info on grep.

pipe(|): Pipe is used to combined more than one command. i.e output of one command is used  as a input for another command. This will help to make multiple commands in a single line. If you want to find list of names from a file and sort them by removing duplicates. You can simply do like this 'grep name file_name | sort -u'

cat: Cat is the command to show or display contents of the file in the console or stdout. Earlier I used vim for seeing content of the file. But now I am using cat along with more command to see the contents. In my opinion this is the best way to see contents. And if you want to process file content, you can use cat along with awk.

awk: This is very power full text processing tool. We can  do text processing efficiently and effectively using this command. Input data for training should be always separated by delimiters like comma(,), pipe(|) or hyphen(-) or mostly CSV format. We can use awk to verify whether input file is formatted properly. You can refer awk use here.

> (redirection): This will help to save intermediate files in a separate file. For example, if you want to save sorted names after searching, you can run this command 'grep name file_name | sort -u > new_filename'. This will save results in a file new_filename.

wc : This command will show total no.of lines, words and characters in a file. I never used this command even though I knew it until I started working for ML project. If you are working on large size files like machine learning projects where required input data is large in size, it is always good verify the file size or no.of lines. In that case this command will really helps a lot. You can find total no.of unique names after grep by combining this command with grep and sort as 'grep name file_name | sort -u|  wc'. If you want to know only lines or characters or words just use 'wc -l filename' for total no.of lines from filename, for words use 'w' and 'c' for characters instead of 'l'.

more: This command is used to see the file content page by page if the content is more than one page. This will really help in seeing and analyzing the content of the file very quickly. And we can also use this command to see the results of a another command by pipe to analyze the results. For example, I want to see uniquely sorted names after grep, I can use like 'grep name file_name | sort -u | more' which will show results in page by page format.

head/tail: Some times its always helpful to see first or last lines of the file. As the name says, 'head filename' will show first ten lines and 'tail filename' will show last 10 lines of the file by default. If you want only first or last three liens you can use 'head -3 filename' or 'tail -3 filename'. You can replace '3' with any numeric if you want. You can also use this with other commands using pipe(|) like 'grep name file_name | sort -u| head' or 'grep name file_name | sort -u | tail'

file: 'file file_name' command will tell the format of the file file_name. Some times filename doesn't contains extension or may be file contains incorrect file extension. In that case you can use this command to know the file format. This is my favorite command. I use this regularly to know file format. One of my personal experience was when I started working in iOS development. I really don't know about 'ipa' file format. I learned by using this file command that ipa is a zip file. You can see related post here. If you are working in machine learning project where file size going to be large, you cant open the file. This command will help in that scenario.

ls -h: I know many of you aware of this command and you might have already used 'ls -ltr' many times. But I am going tell one usefull option is reading the file size quickly. By using '-h' option, ls command will disply the file size in human readable format. This is also one of my favorite command and it is very very help full when you are working CLI.

Happy Learning!!!

Key points to better understand Machine Learning(ML) projects!!!!!

Problem statement: Understanding problem statement is one of the key to get good Machine learning model. Domain knowledge makes crucial role in understanding problem statement.Defining problem statement is not easy. All  the examples or problems available over the web are clearly defined their problem statement. But when you start working on real time use cases, its very hard to understand the problem. Data scientist and Data Analyst will play major in this.

Data: Now a days we are having lot of data in the form of text, images, audio and video format. But the problem with this data is, all this data is unstructured format and not clean.All the sample data available over web (Kaggle for example) is already cleaned data. For practicing or for learning this will help. But when you started working on real time projects, you wont get cleaned data. Data Engineer will play major role in cleaning unstructured data, which is commonly known as pre-training stage.

Understanding Data: Even though cleaned data is available, To get better training model, need to understand data samples clearly. How data is a distributed and what features needs to take from that data.If the data is not distributed equally, ML model will not work properly. Always make sure that input data is equally distributed. Data Analyst will play major role in this.

Happy Learning!!

Useful commands for data cleaning in Machine Learning(ML) training - Part 1

If you are a CLI user , some of these commands may be familiar to you. The hardest part in whole Machine learning(ML) is providing clean data to train. ML is all about data and there is lot of data available these days and huge data is generating every data in the form of text, videos, images etc. All this data is not structured and that is one of the biggest problem. How to make unstructured into a usable format? By using combination of below commands, we can clean raw data by removing noise and make a structured data which further can be used for training a model.

• grep
• pipe(|)
• cat
• awk
• > (redirection)
• wc -l
• more
• head
• tail
• ls -lh
• sed

grep: is a command to find or search for a particular pattern in a given file. you can find more on this.
pipe(|): pipe alone we cant use. But this will be used to join more than one command.
cat: is a command to show contents of a given file in the console.
awk: is very powerful command line utility for text processing.
>(redirection): is the utility to redirect results of any command from console to any other file
wc: This command will give total lines, words and characters from a given file. -l will give only total lines from a given file. This will be very useful to know how big the data file is.
more: this is to see the contents of a given file in pagination format if file content is more than one page. Initially I used to use vi to see the contents of a file, after knowing about more command, I started using more instead of vi and it is very effective as well.
head: to see sample content(first 10 lines) from a given file. you can use -n to specify how many lines you want to show on the console
tail: same as head, but from bottom. To see sample content(last 10 lines) from a given file. you can use -n to specify how many lines you want to show from bottom on the console
ls -lh: Many of us know about ls command. trick here is l and h. l will show file details like permissions and owner etc, h will show the size of the file in human readable format. like if file size is 1424, ls -h will show 1.4K. you can easily tell later one is more readable.
sed: is a stream editor. this command mostly I used for replacing a pattern in the same file.

I write next post on these commands in details.

Happy Learning!!!

How to validate data file is formatted properly before training ML model!!

In machine learning(ML), clean data for training  is the key for a better model.
Must remove noise(unwanted data like special characters) from input data to get clean clean data.
.To do this one of the first step is to make sure data format is consistent in entire input file.
One of the best way to check this is below command. This command should return
Only one value, if not your data file is not properly formatted.

cat file_filename | awk -F’,’ ‘{print NF}’ | sort -u

Let’s see in detail about this command. This command contains below sub parts

cat: is the command to display contents of a text file
awk: this is very power full text processing tool.
sort:this command is to sort and u is for unique and removing duplicates
NF: is number of fields. This will be used In AWK
Pipe(|): this is also very power full command line utilitiy which combines more than one command.

Let’s take a sample file which contains name, age and city of some customers separated by comma, which means each line should contain only three fields separated by comma. But if you see in the below data, in the second line it contains four fields separated by comma.This is very very common scenario in Machine learning training data. And checking this format in a large file not easy. Using above command we can easily verify it.

Chandra, 35,Singapore
Sekhar,Chandra, 26,Guntur
Sekhar,35,Singapore

Applying above command on this text will return the results 3 and 4 as shown below as input data is not consistent. This is due to second line contains one extra comma in it.

After removing extra comma from the second line, that command returns only one value which is 3 as shown below. This return value may vary basing on number of fields in each line, but each line should contain same number of fields which makes this command returns only one value.

cat command will display the contents of the file and redirect to AWK because of pipe and awk will split each line with comma as -F is a field separator and NF will return all the fields after split each line. In this case we have three lines in that first and third lines has three and second line has 4 after split using comma. Sort will show the sorted result with duplicates and -u will show only unique values.

This command will work efficiently on very large files as well. I tried on data files whose file size is around one million lines and it returned results in seconds.


Happy Learning!!!

GREP unix command useful tips!!!

Recently I learned  useful tips from my manager for 'grep' which helps a lot when troubleshooting the  issues with logs. In this post I will explain about below

• How to search multiple strings in a file at the same time.
• How to display found string in the color.
• How to exclude particular string in the search result
• How to display total no.of lines in the grep result

Search a string in a file using grep: Check this link for basic usage  of grep command.  

Lets see this tip with example. Below is the content of a file demo.txt and this file name we are going to use in this example as well.

How to search multiple strings in a file:
In the below screen we are search for the strings 'demo', 'show' and 'multiple' in the file demo.txt and and the result is as follows.

How to display found string in color:
In the above screen, even though grep found the patterns, it is difficult to identify in which line these patterns available. For that you can use grep command property 'color' as shown below.

From the above screen, we can easily identify the found strings as they are highlighted in red color. This color utility will save lot of time when you are searching in debug logs while troubleshooting.

How to exclude particular string in the search result: GREP command will supports option to exclude particular pattern from the result. This is basically not including a string in the result.

In the above screen, initially search for the strings 'demo' and 'show' and in the results I want to exclude the string 'multiple'.

How to display total no.of lines in the grep result: We can use -c option to get the total number of lines of the grep result. If the pattern is unique in each line, that count will be total no.of occurrences of the pattern.

In the above screen, searching for the patterns 'demo' and 'show' results two lines and using -c option will show the total count to two. If the grep results are more, this count option will be very useful.


Happy Learning!!!

Uploading a file using Multer!!

Recently I got a  requirement of uploading a file. We are using NodeJS server, so I explored on NPM and found out very simple library called Multer. It is very simple to use, so sharing here.
                   For uploading a file, we need to know the location where we are going to store the file. For that we need to specify the location for the Multer as shown below. One parameter is destination in the line 20 where we need to specify the location in this case uploads folder in the current directory. Another parameter is filename un the line 23 with which we are going to store in the location. If filename not specified, random number will be given to that file name. 
                   After specifying destination and filename parameters, need to map these Multer storage fields to Multer as shown below in the line 29.

Now let's see how to upload file. While using uploading POST request, need to match the query which is coming in the request parameters as shown below in line 48. In this api , I am using 'file', so query parameter should be 'file', you can use any name, but i used file here. In the line 48, we are using upload.single('file'), upload is actually Multer configured storage in the above image line 29. If file uploading is successful, we will get file name in the call back as shown in the line 49. If file upload fails, this filename will be undefined.

If you want to validate the request before start uploading, just add the method in the POST request as shown below in the line 48. In this case I used a method name as validate, you can use whatever function name you want or even you can skip this validation if you don't want, in that case you just remove that validation method from the line 48.

Now let's see Downloading file. For downloading a file, there is no dependency with Multer, but as we are discussing about uploading a file, lets see downloading file as well.

For the complete code. Refer my github link here.

Happy Learning!!

What is functional programming?

                                       Functional programming (FP)is one of the programming style like procedural or object oriented programming. It will follow Declarative programming paradigm in which, programming will be done using expressions instead of statements. Where as Imperative programming language uses statements(e.g C, Java). Below are the some of the functional programing concepts.

• Pure functions
• No Side effects
• No shared state
• No Mutating state
• Function composition

Pure Functions: A pure function is a function which will return always same value for the same input. The result of a pure function is always depends on input values and its internal logic. Pure function will not read/write anything from outside of that function.  A pure function will eliminate side effects and maintains referential transparency. A pure function can be referential transparency if function call can be replaced with its result.  Below are some of the examples.

• 'chandu'.length is always 6, so its pure function. And instead of 'chandu'.length, you can use 6 as well, so this is also a referential transparency.
• Max(3,7) is always 7, so Max is a pure function
• sqrt(9) is always 3, so its a pure function. 

No Side effects: As stated in pure functions, there will not be any out side change  due to a particular function call. Should avoid shared variables, global variables to eliminate side effects.

No Shared State: Shared state is sharing variable or object or memory location. In functional programming there will not be any shared variable or object. If shared state is there, it violates pure function definition as some other function may change shared object.  If we eliminate shared state, there wont be any change in the order of the function calls as well.

No Mutating state: A mutable object can be modifiable after its creation. In functional program, modification of an object is not allowed after its creation. So that there wont be any side effects. SO Immutability is the another feature of functional programming.

Function composition: Composition is a combination of two or more functions to make a new function. As stated earlier Functional programming is declarative language, so it uses expressions instead of statements. This function composition is very useful to make expressions using functions.

Functional programming languages are used mostly on mathematical calculations and pattern matching and AI. Haskell and Lisp are some of the FP languages. JavaScript and Python will follow some of the FP features.

Happy Learning!!                                                                                          References

What is Apple file system(APFS)!!!

Apple announced its new file system called Apple File System(APFS) at WWDC2016. Lets see what are new features in APFS and why apple introduced this new file system.

Till now apple is using file HFS+(Hierarchal file system).  The problem with this file system is, it designed almost three decades back and it designed basing on the existing drives(HDD's, FlopDisks) on that time. Though HFS+ supports latest SSD, Flash drives, it is not efficient enough. To support all these, Apple introduced new file system called Apple File System. Below are the some of the key features of APFS.

• To make single file system for all apple OS (macOS, iOS, watchOS and tvOS)
• To support and take advantage of SDD and Flash memory disks
• 64-bit support 
• Encryption first
• Space sharing- if partition the disk, this file system will automatically adjusts the memory if one partition has no space
• Feel fast access by reducing the latency -  as SSD and flash disks wont have any spinning needle to read/write unlike HDD's.
• Snap shot and clones - copying and moving files are quick
• Less OS Space

This file system will be available from iOS 10.3, macOS 10.12.4. When users upgrades to 10.3 it version, it will automatically converts file system from existing HFS+ to new Apple File system. Though this conversion will not effect your data, its advised to take the back up before upgrading iOS 10.3 

Happy Upgrading!!!

What is Optional in Swift!!!

                       Apple's new programming language language swift is a very safe language. It will try to make sure that your code is not going to crash. To do this swift provides a feature called optional type. This optional type will store wrapped value if present and nil if no value presents. That basically means any optional variable either contains a value or nil. Lets see deep into optional type. The Optional type is a enumerated value with two values nil and some value which are represented as below.

• Optional.none
• Optional.some(value)

How to use optional: There  are two ways to  use - long form and short form. Mostly short form will be used, but we will see here both for better understanding purpose. Short form is represented by post question mark ?, and long form is represented by Optional key word.

let shortForm: Int? = Int("77")
let longForm: Optional = Int("77")

Optional Binding: We can use optional variable by unwrapping  the value, so that there wont be any runtime error. To unwrap conditionally we have three options

• if let
• gaurd let
• switch

Optional Chaining: Optional chaining is a process for calling methods and properties on a optional that could be nil. if the optional contains a value, it will succeeds and proceeds for the next value, if the optional is nil, it simply returns nil. multiple optional method callings can be chained together and the entire chain will be failed gracefully if any one of the value is nil.

if let someResult = someValue?.someMethod()?.someAnotherMethod()
{
    print("Success")
} else {
    print("failed")
}

Nil-Coalescing Operator: This will be used for the optional to set default value for the nil optional value. And this can be used by doubel question mark ??. This can be also used as chaining

let someResult = someValue?? anotherValue

In this case, someResult will be someValue if it has value and anotherValue if someValue is nil.

Chaining example:

let someResult = someValue?? anotherValue?? anotherValue1

Unconditional Unwrapping: If you are sure that optional has a value,then this unconditional unwrapping will be used  by specifying  forced unwrap operator (postfix !). The problem with this feature is, if the optional value is nil, you will get run time error or possibly your app may crash.

let number = Int("77")!
print(number)
// Prints "77"

It is also possible to use chaining using postfix !.

let isPNG = imagePaths["image"]!.hasSuffix(".png")
print(isPNG)
// Prints "true"

Happy Swifting!!!

References:
Apple Doc

Difference between class and struct in Swift!!!

Structures and classes in swift follow the same syntax for variable, constants and methods. And below are the common things in both class and struct.

• Properties
• Methods
• Initializers to initialize values
• Conform to protocols

And the main differences are

• Class supports inheritance and structs not
• Classes are reference types and structs are value types

Value Type:  When copying or assigning one value type variable to another value type variable, whole data will be copied and they are entirely two new objects. If any change in one object doesn't effect another object. Structures are value types.

//struct definition
struct Name{
    var first = "Pasumarthi"
    var last = "Chandra"
}

//struct variable initialization
var name = Name()

//copying name objet to anotherName object
var anotherName = name

//modifying
anotherName.last = "Chandra sekhar"

print("\(name.first) \(name.last)")
print("\(anotherName.first) \(anotherName.last)")


Result:
Pasumarthi Chandra
Pasumarthi Chandra sekhar

In the above swift code snippet, I have created structure object name and which is assigned to another structure object anotherName and modified anotherName property last, and displayed both name and anotherName objects and both displayed different values.

Reference Type: When copying or assigning one reference type variable to another reference type variable, both will be pointing to the same object and if any change in one object will effect the another object as both are pointing to the same reference. Classes are reference types.

//Class definition
class NameAsClass{
    var first = "Pasumarthi"
    var last = "Chandra"
}

//class initialization
var nameAsClass = NameAsClass()

//Copying one class object to another class object
var anotherNameAsClass = nameAsClass

//modifying 
anotherNameAsClass.last = "Chandra sekhar"
print("\(nameAsClass.first) \(nameAsClass.last)")
print("\(anotherNameAsClass.first) \(anotherNameAsClass.last)")

Result:
Pasumarthi Chandra sekhar
Pasumarthi Chandra sekhar

In the above swift code snippet, I have created class object nameAsClass and which is assigned to another class object anotherNameAsClass and modified anotherNameAsClass property last, and displayed both nameAsClass and anotherNameAsClass objects and both displayed same values.

References:
Click


Happy Coding!!!!

How to write safe and secure code!!!

                                 Coding is very easy task to do, but writing safe and secure code is difficult. In this post I will try to explain some of the rules/steps to make safe and secure code. If we follow these steps we can eliminate most of the failures in our software. As a coder, we need to find out all possible failure cases first and handle them. Some where I read statement like "A developer is like a cab driver in India who sees both sides in one way road". I think this statement is very true. In software application anything can happen, no software is secure and any software can crash at any time due to some simple mistake in the code. So to avoid all simple and silly mistakes and making more safe and secure code, below are the rules with no specific order you need to follow without fail.

• Static Code Analysis(Static analyzer)
• Test Driven Development(TDD)
• Code Review
• Pair Programming

Static Code Analysis: Basically what static analysis will do is, it just scans our code and find out possible errors. There could be some copy-paste errors, some human errors etc. All these can be identified by Static analyzers. Static analysis done by static analyzer which is simply another software which can scan our code and generate report with all errors and warnings.There are lot of open source and commercial static analyzers available on the web. Here are some of simple errors which  identified by static analyzer.

Test Driven Development(TDD): In TDD, first instead of writing code for functionality, need to write all possible test cases for that functionality. After finishing all test cases, run those test cases once. All these test cases will fail as there is no code available for the functionality. Now start writing the code to pass all these test cases. Believe me, It helps a lot in eliminating most of the bugs in the initial stage. Writing test cases for existing code is difficult. So always start tests cases before writing actual functionality. It will take some extra time, but it helps a lot.

Code Review: This is one of the traditional way of finding out silly mistake done by developers. Always make sure that your code is reviewed by some one. Some others reviewing your code doesn't mean that you are not good in coding, it eliminates if any mistakes and it boasts your confidence levels if there are no comments :-) So Always go for the code review and don't skip it.

Pair Programming: This is another new way of coding. Most of the developers thinks that if they are alone, they can write code quickly and efficiently. Yes that is true. But occasionally do pair programming. If possible code with new developer and some times with senior developer. While doing pair programming, basically two developers are seeing that code and two brains are working right!!. In this case, if any mistakes done, another developer identifies it and s/he may give another better way of writing the same code.

                        Till now I have not specified any secure programming techniques rite? If you follow above rules, you can easily eliminate lot of common security related issues. All these steps are not specific to any particular programming language. In whatever  language you are going to write, always follow these steps. Nowadays we have lot of IDE's (like XCode, Eclipse) which are supporting inbuilt frameworks to support static analyzers and TDD.

Enjoy Coding!!!
Happy Coding!!!


References:
Click

What is Internet of Things (IoT)?

                         Now a days Broadband Internet or WiFi is available almost everywhere. So connecting our smart phone to internet is becomes easy due to availability of internet. We can find at least three devices like laptop, smart phone, tab in home and which are connected to WiFi. How about connecting Fridge or washing machine to WiFi? Idea of inter connecting all these devices is Internet of Things or shortly IoT.

                          IoT in simple way is connecting  internet enabled devices such as smart phone, fridge, washing machine or car etc.. Its a relation between people - people, things-things and people - things. These days each person on average using two internet enabled devices like smartphone or smartwatch or laptop. I see in future it will increase to five or six. In future every person on average is going to use five or six devices which can be connected to WiFi. So using IoT, we can connect Smart phone, Laptop, tab, washing machine, fridge etc ..

                              There is a little concern about security and privacy about our personal information as these interconnected devices will share lot of our personal Data. But still IoT is in beginning stage,  there wont be any problem if IoT uses good security methodologies to protect personal data.

References regarding IoT:

https://en.wikipedia.org/wiki/Internet_of_things
http://www.forbes.com/sites/jacobmorgan/2014/05/13/simple-explanation-internet-things-that-anyone-can-understand/
https://www.theguardian.com/technology/2015/may/06/what-is-the-internet-of-things-google

Happy Learning!!!

Archive upload failed due to the issues listed below!!!

When I tried to create a archive in Xcode 7.3 and to upload IPA for TestFlight. I got this error. I tried it multiple times and got the same error. I googled for it and got simple solution and it worked!!. And the solution is trying it after some time. Yes doing it after some time worked for me.

This may be due to network issue or Apple server issue or some thing else. But It works if you try after some time. 

Duplicate message reading from SQS - AWS !!

Recently we faced one issue in reading messages from SQS in AWS cloud where we are processing same message multiple times. This issue we identified by using messsage identifier(mid) of the each message in AWS REDSHIFT table column.

How our system works:

1. Post messages to AWS SQS using one task
2. Read batch of messages from SQS and start processing json message
3. Validate each message in a batch and put in AWS S3 bucket
4. Load into AWS REDSHIFT database
5. Delete batch of messages from SQS after succesfuly processing messages

As I mentioned earlier we faced an issue of some of the messages processing multiple times and this was happening in only one environment and not all environments. To find RCA for this, it took almost three days and below is the RCA. Possible reasons for this to occur is,

Reading a message from SQS and not deleting - if this is the case messages never deletes from SQS and all messages should process multiple times. But this is not happening. Messages are deleting but some messages are processing multiple times
Reading same message by multiple tasks to process - This is happening in our scenario.
Task1 reading batch of messages from SQS and before deleting all theses messages some other task picking some messages from task1 read messages. This is due to visibility time out of message in the SQS. Lets see how this happens.

                                 When a task read batch of messages from SQS , these messages are moved to inflight mode and not visible to other task to read as these messages are under processing. And SQS has a property called visibility time out, so message in the inflight mode message are not visible to other tasks until this time out completes.

                                     Before this time out expires, we need to complete our process of message validation, loading to S3, storing to REDSHIFT and deleting. In our case some of the messages are not completing this process(or not deleting) with in the visibility time out(10sec in this case). So causing the message visible again after the visibility timeout expires.

                                     As I mentioned this was happening only one environment because, in that environment only we have visibility time out 10secs and all other enviroments we have 20secs. Our task is not completing with in 10secs and causing the duplicate message processing issue. To solve this issue, we just increased visibility time out to 20secs.

I hope this helps.

Happy Reading!!!

Some of use full Programming Books!!

Below are some of use full programming books and sites. 

Programming books:

Apprenticeship Pattern

Learning JavaScript Design Patterns

Introduction to Computing

97 Things Every Programmer Should Know

The Codeless Code

How to Design Programs

Foundations of Computer Science

Producing Open Source Software

Foundations of Programming

The Nature of Code

Patterns of Software

The Architecture of Open Source Applications

How to Think Like a Computer Scientist

Learn Regex The Hard Way

The Little Introduction To Programming

Building Skills in Programming
Uncle Bob Clean Code


Happy Coding!!!


Reference: From Here

S3 load errors in Redshift(AWS) COPY command

We have faced lot of weird issues while loading S3 bucket files into redshift. I will try to explain all issues what we faced. Before going that , lets see what  are Valid S3 file should contain

• No.of values in S3 bucket are exactly equal to the no.of columns in the redshift table
• Each value in S3 separated with a delimiter, in our case its pipe(|)
• Each line in S3 file is exactly one insert statement on redshift
• Empty values will be passed in the S3 file for corresponding optional field in table

To store S3 file content to redshift database, AWS provides a COPY command  which stores bulk or batch of S3 data into redshift.
Lets assume there is a table testMessage in redshift which has three columns id of integer type, name of varchar(10) type and msg of varchar(10) type.

S3 file to redshift inserting COPY command is below

copy testMessage (id, name, msg) from 's3://blogpost.testbucket/test/file.txt' credentials 'aws_access_key_id=;aws_secret_access_key=;token=' delimiter '|' ACCEPTINVCHARS '_'

To insert values from S3 file, sammple S3 file could be

77|chandu|chanduthedev


In this file total values are three which is equal to no.of columns in the  testMessage table columns and each value separated by pipe(|) symbol.

Lets see another S3 sample file
88||chanduthedev

In this file, we have one empty value for name column in table testMessage in redshift. So far so good. Lets take some S3 files which cause to fail redshift COPY command

99|chanduthedev

In this S3 file contains only two values 99 and chanduthdev, and missing third value which causes to file S3 load COPY command

99|ch
and u|chanduthedev

In this file, second value is ch\nand u which conatins new line(\n) characters, so it becomes two rows in the S3 file which means two insert statements to redshift COPY command and. First row becomes two value insert statments which is invalid and second one is another two value invalid statement.

For these invalid S3 file you may get below error message.

Load into table 'testMessage' failed.  Check 'stl_load_errors' system table for details
and in AWS you may get below error
Delimiter not found 

Lets take another failure S3 file which has delimiter as value for name column

77|chan|234|chanduthedev


In the above S3 file, it looks 4 values because of extra pipe(|) character for the name chan|1234 which causes redshift COPY command to treat S3 file has four values, but table has three values.

For S3 load failures, the most common reason could be special characters or escape characters like new line(\n), double quotes("), single quotes etc. Either you need to escape those special characters or remove those special characters.

We followed later idea of removing special charasters while processing and storing in the redshift. But later came to know that we can use ESCAPE key word in COPY command.


copy testMessage (id, name, msg) from 's3://blogpost.testbucket/test/file.txt' credentials 'aws_access_key_id=;aws_secret_access_key=;token=' delimiter '|' ACCEPTINVCHARS '_' ESCAPE

adding ESCAPE to COPY command will solve lot of these issues. So always check for ESCPAPE

Happy Debugging...

String length exceeds DDL length - S3 bucket Load error on redshift(AWS)

We have recently faced one tricky issue in AWS cloud while loading S3 file into Redshift using python. It took almost whole day to indentify the issue and fixing it.

Our way of doing things in AWS cloud as below

1. Get the json message from SQS using python
2. Validating the fields from json message received in step 1
3. Make a csv format after validating and store file in S3 bucket in AWS using csv python library
4. Load S3 file into AWS redshift database using copy command

The above process in simple terms, read the message, proces it and insert into redshift Database. In this process there could be a chance of failures like

1. While validating json message from SQS, we may get invalid input which python cant identify like escape characters - this also we faced and I will make another blog post on this soon
2. While write to s3 file we may get some extra escape characters - This problem we faced and I am covering now
3. While inserting into redshift db 
1. if we try to insert invalid datatype value in the column (for integer column trying to insert varchar value)
2. if the inserting value exceeds the length of the column (like for msg column lenght is 10, if we try to isnert more than 10 chars it will fail) - this was due to step 2
3. if S3 file does not have proper delimiters

 For simplifying big problem, I am assuming there is one table testMessage in redshift which has two columns id of integer type and msg of varchar(10) type.
To insert the values into testMessage table using above process, we are expecting a json message which contains id and msg keys. Sample message shown below

{"id":7, "msg":"testfile"}

As per the above four step process

1. json message  from SQS
• {"id":7, "msg":"testfile"}
2. Validating key - values in the json message
• looks valid as id field contains integer and msg field contains string less than or equal to 10 chars
3. CSV format S3 file
• id|testfile
4. Loading to redshift
• copy testMessage (id, msg) from 's3://blogpost.testbucket/test/file.txt' credentials 'aws_access_key_id=;aws_secret_access_key=;token=' delimiter '|' ACCEPTINVCHARS '_' ESCAPE

This will work fine as there are no validation failures and no sepecial characters in the message. Lets take another message which contains special characters like double quotes

{"id":7, "msg":"\"testfile\""}

1. json message  sqs message
• {"id":7, "msg":"\"testfile\""}
2. Validating key - values in the json message
• looks valid as id field contains integer and msg field contains string less than or equal to 10 chars
3. CSV format s3 file
• id|""testfile""
4. Loading to redshift
• copy testMessage (id, msg) from 's3://blogpost.testbucket/test/file.txt' credentials 'aws_access_key_id=;aws_secret_access_key=;token=' delimiter '|' ACCEPTINVCHARS '_' ESCAPE

Bhooom .... 
Loading to redshift copy command fails with error message 'String length exceeds DDL length '. 
Here comes the actual problem we faced!!!!!

If we observe in step3 S3 file, actual msg value is '"testfile"' whose length is 10, but we can find two extra double quotes in the s3 file which causes exceeding the length of the string to 12 from 10.
From where all these extra double quotes has come??
Until validations and processing message everything looks fine, but while writing into s3 file with CSV format, CSV library in python adds extra escape characters which causes actual problem.

In our case we have double quotes which is a special character, and csv library  adds another double quote as escape character which increase length from 10 to 12 which causes the problem

To avoid this problem, we can use

csv.register_dialect(dialect, doublequote=False, escapechar='\\', quoting=csv.QUOTE_NONE)

This means we are making doublequotes as false and treating escape characters are empty.

1. escapechar='\\' this specifies, no escape characters are there, treat all special characters are normal characters.
2. escapechar='' this doesn't mean no escape characters, which means empty as escape character

to pass empty value as escape characters use point 1 and not point 2. If you use you will get below error which means its expecting a escape character, but you have not specified any value

Error: need to escape, but no escapechar set

This fix looks simple in this example, but in real time scenario we have almost 40 columns in a table and the SQS json message was more than 7000(7K) characters with lot of special characters like newline(\n), carriage return(\r), double slashes(\\), double quotes(") etc .... We worked a lot to identify this issue and sharing here to help others.


Hope this helps ...
Happy Debugging ......

JSON vs XML which one is better?

JSON
  - good for web services
  - human readable format
  - it uses javascript eval() function which may cause some security flaw as we can excecute the js object

XML
  - good for configurations
  - Using XPath path, we can directly access the element
  - Using XSLT template, we can easily convert from Xml to json, csv or any format


You can go with either JSON or XML basing on your requirement. Many developers think that JSON object is a light weight and which is better than XML. But in my opinion both are same.