Tuesday, December 6, 2016

What is Optional in Swift!!!

                       Apple's new programming language language swift is a very safe language. It will try to make sure that your code is not going to crash. To do this swift provides a feature called optional type. This optional type will store wrapped value if present and nil if no value presents. That basically means any optional variable either contains a value or nil. Lets see deep into optional type. The Optional type is a enumerated value with two values nil and some value which are represented as below.
  • Optional.none
  • Optional.some(value)
How to use optional: There  are two ways to  use - long form and short form. Mostly short form will be used, but we will see here both for better understanding purpose. Short form is represented by post question mark ?, and long form is represented by Optional key word.

let shortForm: Int? = Int("77")
let longForm: Optional = Int("77")

Optional Binding: We can use optional variable by unwrapping  the value, so that there wont be any runtime error. To unwrap conditionally we have three options
  • if let
  • gaurd let
  • switch
Optional Chaining: Optional chaining is a process for calling methods and properties on a optional that could be nil. if the optional contains a value, it will succeeds and proceeds for the next value, if the optional is nil, it simply returns nil. multiple optional method callings can be chained together and the entire chain will be failed gracefully if any one of the value is nil.

if let someResult = someValue?.someMethod()?.someAnotherMethod()
} else {
Nil-Coalescing Operator: This will be used for the optional to set default value for the nil optional value. And this can be used by doubel question mark ??. This can be also used as chaining

let someResult = someValue?? anotherValue

In this case, someResult will be someValue if it has value and anotherValue if someValue is nil.

Chaining example:
let someResult = someValue?? anotherValue?? anotherValue1

Unconditional Unwrapping: If you are sure that optional has a value,then this unconditional unwrapping will be used  by specifying  forced unwrap operator (postfix !). The problem with this feature is, if the optional value is nil, you will get run time error or possibly your app may crash.

let number = Int("77")!
// Prints "77"

It is also possible to use chaining using postfix !.

let isPNG = imagePaths["image"]!.hasSuffix(".png")
// Prints "true"

Happy Swifting!!!

Apple Doc

Monday, December 5, 2016

Difference between class and struct in Swift!!!

Structures and classes in swift follow the same syntax for variable, constants and methods. And below are the common things in both class and struct.
  • Properties
  • Methods
  • Initializers to initialize values
  • Conform to protocols
And the main differences are
  • Class supports inheritance and structs not
  • Classes are reference types and structs are value types
Value Type:  When copying or assigning one value type variable to another value type variable, whole data will be copied and they are entirely two new objects. If any change in one object doesn't effect another object. Structures are value types.

//struct definition
struct Name{
    var first = "Pasumarthi"
    var last = "Chandra"

//struct variable initialization
var name = Name()

//copying name objet to anotherName object
var anotherName = name

anotherName.last = "Chandra sekhar"

print("\(name.first) \(name.last)")
print("\(anotherName.first) \(anotherName.last)")

Pasumarthi Chandra
Pasumarthi Chandra sekhar

In the above swift code snippet, I have created structure object name and which is assigned to another structure object anotherName and modified anotherName property last, and displayed both name and anotherName objects and both displayed different values.

Reference Type: When copying or assigning one reference type variable to another reference type variable, both will be pointing to the same object and if any change in one object will effect the another object as both are pointing to the same reference. Classes are reference types.

//Class definition
class NameAsClass{
    var first = "Pasumarthi"
    var last = "Chandra"

//class initialization
var nameAsClass = NameAsClass()
//Copying one class object to another class object
var anotherNameAsClass = nameAsClass
anotherNameAsClass.last = "Chandra sekhar"
print("\(nameAsClass.first) \(nameAsClass.last)")
print("\(anotherNameAsClass.first) \(anotherNameAsClass.last)")

Pasumarthi Chandra sekhar
Pasumarthi Chandra sekhar

In the above swift code snippet, I have created class object nameAsClass and which is assigned to another class object anotherNameAsClass and modified anotherNameAsClass property last, and displayed both nameAsClass and anotherNameAsClass objects and both displayed same values.


Happy Coding!!!!

Thursday, December 1, 2016

How to write safe and secure code!!!

                                 Coding is very easy task to do, but writing safe and secure code is difficult. In this post I will try to explain some of the rules/steps to make safe and secure code. If we follow these steps we can eliminate most of the failures in our software. As a coder, we need to find out all possible failure cases first and handle them. Some where I read statement like "A developer is like a cab driver in India who sees both sides in one way road". I think this statement is very true. In software application anything can happen, no software is secure and any software can crash at any time due to some simple mistake in the code. So to avoid all simple and silly mistakes and making more safe and secure code, below are the rules with no specific order you need to follow without fail.

  • Static Code Analysis(Static analyzer)
  • Test Driven Development(TDD)
  • Code Review
  • Pair Programming

Static Code Analysis: Basically what static analysis will do is, it just scans our code and find out possible errors. There could be some copy-paste errors, some human errors etc. All these can be identified by Static analyzers. Static analysis done by static analyzer which is simply another software which can scan our code and generate report with all errors and warnings.There are lot of open source and commercial static analyzers available on the web. Here are some of simple errors which  identified by static analyzer.

Test Driven Development(TDD): In TDD, first instead of writing code for functionality, need to write all possible test cases for that functionality. After finishing all test cases, run those test cases once. All these test cases will fail as there is no code available for the functionality. Now start writing the code to pass all these test cases. Believe me, It helps a lot in eliminating most of the bugs in the initial stage. Writing test cases for existing code is difficult. So always start tests cases before writing actual functionality. It will take some extra time, but it helps a lot.

Code Review: This is one of the traditional way of finding out silly mistake done by developers. Always make sure that your code is reviewed by some one. Some others reviewing your code doesn't mean that you are not good in coding, it eliminates if any mistakes and it boasts your confidence levels if there are no comments :-) So Always go for the code review and don't skip it.

Pair Programming: This is another new way of coding. Most of the developers thinks that if they are alone, they can write code quickly and efficiently. Yes that is true. But occasionally do pair programming. If possible code with new developer and some times with senior developer. While doing pair programming, basically two developers are seeing that code and two brains are working right!!. In this case, if any mistakes done, another developer identifies it and s/he may give another better way of writing the same code.

                        Till now I have not specified any secure programming techniques rite? If you follow above rules, you can easily eliminate lot of common security related issues. All these steps are not specific to any particular programming language. In whatever  language you are going to write, always follow these steps. Nowadays we have lot of IDE's (like XCode, Eclipse) which are supporting inbuilt frameworks to support static analyzers and TDD.

Enjoy Coding!!!
Happy Coding!!!


Monday, September 12, 2016

What is Internet of Things (IoT)?

                         Now a days Broadband Internet or WiFi is available almost everywhere. So connecting our smart phone to internet is becomes easy due to availability of internet. We can find at least three devices like laptop, smart phone, tab in home and which are connected to WiFi. How about connecting Fridge or washing machine to WiFi? Idea of inter connecting all these devices is Internet of Things or shortly IoT.

                          IoT in simple way is connecting  internet enabled devices such as smart phone, fridge, washing machine or car etc.. Its a relation between people - people, things-things and people - things. These days each person on average using two internet enabled devices like smartphone or smartwatch or laptop. I see in future it will increase to five or six. In future every person on average is going to use five or six devices which can be connected to WiFi. So using IoT, we can connect Smart phone, Laptop, tab, washing machine, fridge etc ..

                              There is a little concern about security and privacy about our personal information as these interconnected devices will share lot of our personal Data. But still IoT is in beginning stage,  there wont be any problem if IoT uses good security methodologies to protect personal data.

References regarding IoT:


Happy Learning!!!

Monday, August 29, 2016

Archive upload failed due to the issues listed below!!!

When I tried to create a archive in Xcode 7.3 and to upload IPA for TestFlight. I got this error. I tried it multiple times and got the same error. I googled for it and got simple solution and it worked!!. And the solution is trying it after some time. Yes doing it after some time worked for me.

This may be due to network issue or Apple server issue or some thing else. But It works if you try after some time. 

Thursday, August 25, 2016

Duplicate message reading from SQS - AWS !!

Recently we faced one issue in reading messages from SQS in AWS cloud where we are processing same message multiple times. This issue we identified by using messsage identifier(mid) of the each message in AWS REDSHIFT table column.

How our system works:

  1. Post messages to AWS SQS using one task
  2. Read batch of messages from SQS and start processing json message
  3. Validate each message in a batch and put in AWS S3 bucket
  4. Load into AWS REDSHIFT database
  5. Delete batch of messages from SQS after succesfuly processing messages

As I mentioned earlier we faced an issue of some of the messages processing multiple times and this was happening in only one environment and not all environments. To find RCA for this, it took almost three days and below is the RCA. Possible reasons for this to occur is,

Reading a message from SQS and not deleting - if this is the case messages never deletes from SQS and all messages should process multiple times. But this is not happening. Messages are deleting but some messages are processing multiple times
Reading same message by multiple tasks to process - This is happening in our scenario.
Task1 reading batch of messages from SQS and before deleting all theses messages some other task picking some messages from task1 read messages. This is due to visibility time out of message in the SQS. Lets see how this happens.

                                 When a task read batch of messages from SQS , these messages are moved to inflight mode and not visible to other task to read as these messages are under processing. And SQS has a property called visibility time out, so message in the inflight mode message are not visible to other tasks until this time out completes.

                                     Before this time out expires, we need to complete our process of message validation, loading to S3, storing to REDSHIFT and deleting. In our case some of the messages are not completing this process(or not deleting) with in the visibility time out(10sec in this case). So causing the message visible again after the visibility timeout expires.

                                     As I mentioned this was happening only one environment because, in that environment only we have visibility time out 10secs and all other enviroments we have 20secs. Our task is not completing with in 10secs and causing the duplicate message processing issue. To solve this issue, we just increased visibility time out to 20secs.

I hope this helps.

Happy Reading!!!

Saturday, February 20, 2016

S3 load errors in Redshift(AWS) COPY command

We have faced lot of weird issues while loading S3 bucket files into redshift. I will try to explain all issues what we faced. Before going that , lets see what  are Valid S3 file should contain

  • No.of values in S3 bucket are exactly equal to the no.of columns in the redshift table
  • Each value in S3 separated with a delimiter, in our case its pipe(|)
  • Each line in S3 file is exactly one insert statement on redshift
  • Empty values will be passed in the S3 file for corresponding optional field in table

To store S3 file content to redshift database, AWS provides a COPY command  which stores bulk or batch of S3 data into redshift.
Lets assume there is a table testMessage in redshift which has three columns id of integer type, name of varchar(10) type and msg of varchar(10) type.

S3 file to redshift inserting COPY command is below

copy testMessage (id, name, msg) from 's3://blogpost.testbucket/test/file.txt' credentials 'aws_access_key_id=;aws_secret_access_key=;token=' delimiter '|' ACCEPTINVCHARS '_'

To insert values from S3 file, sammple S3 file could be


In this file total values are three which is equal to no.of columns in the  testMessage table columns and each value separated by pipe(|) symbol.

Lets see another S3 sample file

In this file, we have one empty value for name column in table testMessage in redshift. So far so good. Lets take some S3 files which cause to fail redshift COPY command


In this S3 file contains only two values 99 and chanduthdev, and missing third value which causes to file S3 load COPY command

and u|chanduthedev

In this file, second value is ch\nand u which conatins new line(\n) characters, so it becomes two rows in the S3 file which means two insert statements to redshift COPY command and. First row becomes two value insert statments which is invalid and second one is another two value invalid statement.

For these invalid S3 file you may get below error message.

Load into table 'testMessage' failed.  Check 'stl_load_errors' system table for details
and in AWS you may get below error
Delimiter not found 

Lets take another failure S3 file which has delimiter as value for name column


In the above S3 file, it looks 4 values because of extra pipe(|) character for the name chan|1234 which causes redshift COPY command to treat S3 file has four values, but table has three values.

For S3 load failures, the most common reason could be special characters or escape characters like new line(\n), double quotes("), single quotes etc. Either you need to escape those special characters or remove those special characters.

We followed later idea of removing special charasters while processing and storing in the redshift. But later came to know that we can use ESCAPE key word in COPY command.

copy testMessage (id, name, msg) from 's3://blogpost.testbucket/test/file.txt' credentials 'aws_access_key_id=;aws_secret_access_key=;token=' delimiter '|' ACCEPTINVCHARS '_' ESCAPE

adding ESCAPE to COPY command will solve lot of these issues. So always check for ESCPAPE

Happy Debugging...

Friday, February 19, 2016

String length exceeds DDL length - S3 bucket Load error on redshift(AWS)

We have recently faced one tricky issue in AWS cloud while loading S3 file into Redshift using python. It took almost whole day to indentify the issue and fixing it.

Our way of doing things in AWS cloud as below
  1. Get the json message from SQS using python
  2. Validating the fields from json message received in step 1
  3. Make a csv format after validating and store file in S3 bucket in AWS using csv python library
  4. Load S3 file into AWS redshift database using copy command

The above process in simple terms, read the message, proces it and insert into redshift Database. In this process there could be a chance of failures like
  1. While validating json message from SQS, we may get invalid input which python cant identify like escape characters - this also we faced and I will make another blog post on this soon
  2. While write to s3 file we may get some extra escape characters - This problem we faced and I am covering now
  3. While inserting into redshift db 
    1. if we try to insert invalid datatype value in the column (for integer column trying to insert varchar value)
    2. if the inserting value exceeds the length of the column (like for msg column lenght is 10, if we try to isnert more than 10 chars it will fail) - this was due to step 2
    3. if S3 file does not have proper delimiters
 For simplifying big problem, I am assuming there is one table testMessage in redshift which has two columns id of integer type and msg of varchar(10) type.
To insert the values into testMessage table using above process, we are expecting a json message which contains id and msg keys. Sample message shown below

{"id":7, "msg":"testfile"}

As per the above four step process
  1. json message  from SQS
    • {"id":7, "msg":"testfile"}
  2. Validating key - values in the json message
    • looks valid as id field contains integer and msg field contains string less than or equal to 10 chars
  3. CSV format S3 file
    • id|testfile
  4. Loading to redshift
    • copy testMessage (id, msg) from 's3://blogpost.testbucket/test/file.txt' credentials 'aws_access_key_id=;aws_secret_access_key=;token=' delimiter '|' ACCEPTINVCHARS '_' ESCAPE
This will work fine as there are no validation failures and no sepecial characters in the message. Lets take another message which contains special characters like double quotes

{"id":7, "msg":"\"testfile\""}

  1. json message  sqs message
    • {"id":7, "msg":"\"testfile\""}
  2. Validating key - values in the json message
    • looks valid as id field contains integer and msg field contains string less than or equal to 10 chars
  3. CSV format s3 file
    • id|""testfile""
  4. Loading to redshift
    • copy testMessage (id, msg) from 's3://blogpost.testbucket/test/file.txt' credentials 'aws_access_key_id=;aws_secret_access_key=;token=' delimiter '|' ACCEPTINVCHARS '_' ESCAPE

Bhooom .... 
Loading to redshift copy command fails with error message 'String length exceeds DDL length '. 
Here comes the actual problem we faced!!!!!

If we observe in step3 S3 file, actual msg value is '"testfile"' whose length is 10, but we can find two extra double quotes in the s3 file which causes exceeding the length of the string to 12 from 10.
From where all these extra double quotes has come??
Until validations and processing message everything looks fine, but while writing into s3 file with CSV format, CSV library in python adds extra escape characters which causes actual problem.

In our case we have double quotes which is a special character, and csv library  adds another double quote as escape character which increase length from 10 to 12 which causes the problem

To avoid this problem, we can use

csv.register_dialect(dialect, doublequote=False, escapechar='\\', quoting=csv.QUOTE_NONE)

This means we are making doublequotes as false and treating escape characters are empty.

  1. escapechar='\\' this specifies, no escape characters are there, treat all special characters are normal characters.
  2. escapechar='' this doesn't mean no escape characters, which means empty as escape character

to pass empty value as escape characters use point 1 and not point 2. If you use you will get below error which means its expecting a escape character, but you have not specified any value

Error: need to escape, but no escapechar set

This fix looks simple in this example, but in real time scenario we have almost 40 columns in a table and the SQS json message was more than 7000(7K) characters with lot of special characters like newline(\n), carriage return(\r), double slashes(\\), double quotes(") etc .... We worked a lot to identify this issue and sharing here to help others.

Hope this helps ...
Happy Debugging ......

Popular Posts